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1.0  Introduction  and  Background 

1.1  Overview  of  Transformation 

Transformation  is  the  roadmap  that  will  lead  the  U.S.  to  “...a  future  foree  that  is  defined  less  by 
size  and  more  by  mobility  and  swiftness,  one  that  is  easier  to  deploy  and  sustain,  one  that  relies 
more  heavily  on  stealth,  preeision  weaponry  and  information  teehnologies.”' 

The  need  for  transformation  of  the  military  is  driven  by  the  changing  strategic  environment  (9-11 
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and  the  War  on  Terrorism)  that  the  U.S.  faces.  Transformation  is  necessary  because: 

■  U.S.  military  superiority  cannot  be  assumed  in  the  future.  As  Information  Age 
technologies  proliferate,  U.S.  dominance  will  increasingly  be  challenged  in  novel  ways. 

■  Growing  asymmetric  threats  require  new  ways  of  thinking  about  conflict  that  require 
creative  approaches 

■  Force-on-force  challenges  are  likely  to  increase  as  adversaries  seek  to  take  advantage  of 
changes  in  global  power  relations  resulting  from  the  transition  to  the  Information  Age. 

■  Technological  changes  make  transformation  of  the  military  imperative;  there  is  a  window 
of  opportunity  to  leverage  U.S.  competitive  advantage  into  the  future. 

■  The  stakes  are  very  high;  if  the  U.S.  fails  to  transform,  current  superiority  will  be 
increasingly  challenged,  regional  competitors  will  emerge,  and  conflict  will  become  more 
likely. 

Transformation  of  this  magnitude  does  not  occur  in  isolation.  The  transformation  of  the  military 
is,  in  fact,  part  of  the  larger  transition  from  the  industrial  to  the  Information  Age  that  is  occurring 
simultaneously  in  societies  and  economies  around  the  world.  This  transition  is  enabled  by  rapid 
changes  in  technologies  that  precipitate  rapid  coevolutionary  changes  in  strategies,  concepts, 
processes  and  organizations. 

The  Office  of  Force  Transformation  (OFT)  is  chartered  to  take  the  lead  in  moving  the  U.S. 
military  from  an  Industrial  Age  organization  to  an  Information  Age  organization.  It  has 
established  six  operational  goals  to  focus  transformation  efforts: 

♦  Protect  critical  bases  of  operations  (U.S.  homeland,  forces  abroad,  allies,  and  friends) 
and  defeat  CBRNE  weapons  and  their  means  of  delivery. 

♦  Assure  information  systems  in  the  face  of  attack  and  conduct  effective  information 
operations. 

♦  Project  and  sustain  U.S.  forces  in  distant  anti-access  or  area-denial  environments  and 
defeat  anti-access  and  area-denial  threats. 

♦  Deny  enemy  sanctuary  by  providing  persistent  surveillance,  tracking,  and  rapid 
engagement  with  high-volume  strike,  through  a  combination  of  complementary  air  and 
ground  capabilities,  against  critical  mobile  and  fixed  targets  at  various  ranges  and  in  all 
weather  and  terrains. 


*  President  George  W.  Bush.  Defense  Planning  Guidance  (DPG)  April  2003:  pg  3. 
^  Defense  Planning  Guidance:  pg  4-6. 
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♦  Enhance  the  capability  and  survivability  of  space  systems  and  supporting  infrastructure. 

♦  Leverage  information  technology  and  innovative  concepts  to  develop  an  interoperable, 
joint  C4ISR  architecture  and  capability  that  includes  a  tailorable  joint  operational  picture. 

1.2  Network  Centric  Theory 

The  OFT  has  determined  that  Network  Centric  Warfare  (NCW)  is  the  core  concept  that  guides 
the  transformation  of  the  U.S.  military.  NCW  is  the  embodiment  of  Information  Age  warfare.  It 
is  a  new  theory  of  war  based  on  Information  Age  principles  and  phenomena,  and  can  be 
summarized  by  the  tenets.  These  state  that  a  robustly  networked  force  improves  information 
sharing  and  collaboration,  which  enhances  the  quality  of  information  and  shared  situational 
awareness.  This  enables  further  collaboration  and  self-synchronization  and  improves 
sustainability  and  speed  of  command,  which  ultimately  result  in  dramatically  increased  mission 
effectiveness.  Figure  1-1  represents  the  original  articulation  of  the  NCW  Value  Chain. 


Figure  1-1.  NCW  Value  Chain 


As  part  of  the  efforts  to  develop  and  mature  the  concepts  of  Network  Centric  Warfare,  the  Office 
of  Force  Transformation  (OFT)  and  the  Office  of  the  Assistant  Secretary  of  Defense,  Networks 
and  Information  Integration  (OASD/NII)  have  been  collaborating  on  the  development  of  a 
Conceptual  Framework  for  Network  Centric  Warfare/Operations  (NCW/O)"^  and  a  variety  of 
other  NCO  related  research,  outreach,  and  publications.  Together  they  have  developed  a 
Network  Centric  Operations  Conceptual  Framework  (NCO  CF)  for  Assessment  that  identifies 


^  Department  of  Defense.  Network  Centric  Warfare  Report  to  Congress.  July  2001. 

^  The  Terms  Network  Centric  Warfare  (NCW)  and  Network  Centric  Operations  (NCO)  are  used  interchangeably  in 
this  document.  However,  the  latter  term  (NCO)  is  preferred  because  it  implies  correctly  that  the  theory  of  Network 
Centric  Warfare  applies  to  a  much  broader  domain  of  phenomena  and  is  not  limited  to  warfare. 
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key  concepts  and  linkages  to  output  measures  in  the  Network  Centric  Warfare  value  chain  in  the 
context  of  the  physical,  information,  cognitive,  and  social  domains.  The  framework  identifies  a 
vector  of  attributes  for  each  concept  and  defines  important  classes  of  attributes  that  are 
measurable  with  specific  metrics.  The  initial  version  of  the  NCO  Conceptual  Framework  was 
successfully  applied  and  initially  validated  using  an  Air-to-Air  combat  case  study  performed  by 
the  RAND  Corporation.  ^ 

This  document  describes  the  NCO  Conceptual  Framework  for  Assessment  in  detail.  Section  1.0 
is  this  Introduction.  Section  2.0  provides  an  overview  of  the  NCO  Conceptual  Framework  and 
discusses  its  purpose  and  limitations.  Section  3.0  describes  the  top-level  structure  of  the 
framework.  Section  4.0  drills  down  in  the  Conceptual  Framework  and  describes  the  attributes 
and  metrics  for  each  top-level  concept.  Section  5.0  provides  a  summary  and  conclusion. 


^  An  example  of  a  ease  study  template,  adapted  from  the  RAND  Air-to-air  eombat  ease  study,  ean  be  found  at 
http://www.oft.osd. miblibrary/library  files/doeument  235  DRAFT  neo  Template  Air  to  Air.pdf/ 
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2.0  NCO  Conceptual  Framework 

2.1  Overview 

The  Conceptual  Framework  is  being  developed  by  the  Office  of  Force  Transformation  (OFT) 
and  the  Command  and  Control  Research  Program  (CCRP)  of  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Command,  Control,  Communications,  and  Intelligence  (ASD/NII) 
(formerly  ASD/C3I).^ 

The  objective  is  to  develop  a  set  metrics  to  assess  the  tenets  of  NCW  as  presented  in 
Understanding  Information  Age  Warfare  axiA  Network  Centric  Warfare. 

In  order  to  develop  metrics  for  the  tenets,  it  is  first  necessary  to  identify  a  “top-level” 
representation  of  NCO  concepts  and  their  relations.  Once  the  important  concepts  and  their 
relations  are  identified,  one  can  then  “drill  down”  and  identify  attributes  and  metrics  for  each 
concept.  The  “NCO  Conceptual  Framework”  is  the  result  of  that  process.  While  it  provides  a 
means  to  evaluate  NCO  hypotheses,  it  also  clarifies  and  illuminates  important  aspects  of  NCO 
theory  that  were  only  implicit  in  the  original  tenets. 

Figure  2-1  is  the  top-level  Conceptual  Framework. 


Figure  2-1.  The  NCO  Conceptual  Framework 

Figure  2-4  below  represents  the  evolution  of  the  Conceptual  Framework  from  the  original  tenets. 


^  With  support  of  Evidence  Based  Research,  Inc.  and  the  RAND  Corporation. 

’  Alberts,  David  S.,  John  Garstka,  Richard  E.  Hayes,  and  David  T.  Signori.  Understanding  Information  Age  Warfare. 
Washington,  DC:  CCRP  Publication  Series.  2002. 

*  Alberts,  David  S.,  John  J.  Garstka,  and  Fredrick  P.  Stein.  Network  Centric  Warfare:  Developing  and  Leveraging 
Information  Superiority.  2"‘* *  Edition  (Revised).  Washington,  DC:  CCRP  Publication  Series.  2002. 
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Tenets  of  NCW  {DoD  Report  to  Congress  on  Network  Centric  Warfare): 

•  A  robustly  networked  force  improves  information  sharing 

•  Information  sharing  and  collaboration  enhances  the 
quality  of  information  and  shared  situational  awareness 

•  Shared  situational  awareness  enables  collaboration  and  self¬ 
synchronization,  and  enhances  sustainability  and  speed  of  command 

•  These  in  turn  dramatically  increase  mission  effectiveness 


I  Net ‘RgdinKS ofNodK^ 
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NCO  Conceptual  Framework  (2003) 


Figure  2-4.  NCO  Framework  Evolution 
The  NCO  Conceptual  Framework; 

■  Builds  on  the  tenets  of  NCW 

■  Is  best  understood  as  a  generic  “process  model” 

■  Explicitly  recognizes  the  key  role  of  the  “social  domain” 

■  Incorporates  important  research  on  “sensemaking” 

■  Identifies  key  concepts  important  in  most  workflow  processes 

■  Identifies  potential  dependencies  among  concepts 

■  Identifies  and  defines  Attributes  and  Metrics  for  each  concept 

■  Is  scalable  across  different  levels  of  aggregation 

■  Provides  a  basis  for  quantitative  exploration  and/or  assessment  of 

o  NCW  hypotheses 

o  Investment  strategies  and  other  DOTML-PF  related  issues 

2.1.1  Innovations  of  the  Conceptual  Framework 

Network  Centric  Operations  is  not  about  hardware  and  routers — it  is  about  people,  organizations, 
and  processes.  The  Conceptual  Framework  highlights  the  fact  that  network  centric  operations 
cut  across  several  domains:  physical,  information,  cognitive  and  social.  The  central  role  of 
social  interactions  (including  collaboration)  is  evident  in  the  Conceptual  Framework.  While  the 
original  NCO  work  highlighted  the  physical,  information,  and  cognitive  domains,  the  most 
recent  work  introduces  the  social  domain  as  an  important  element.  The  framework  also 
distinguishes  between  individuals  and  “groups”  (teams,  organizations,  etc.).  This  is  an  especially 
important  innovation  as  future  operations  are  expected  to  be  joint  and  involve  interagency 
coordination  and  international  partners. 
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Development  of  the  Coneeptual  Framework  also  led  to  the  emergenee  of  Agility  as  an  especially 
important  concept  for  Network  Centric  Operations.  Agility  captures  the  essence  of 
transformation  and  is  highlighted  in  the  top-level  diagram  as  C2  Agility  and  Force  Agilily. 
Agility  refers  to  the  ability  to  be  robust,  flexible,  responsive,  innovative,  resilient,  and  adaptive.^ 

An  important  innovation  that  emerged  is  the  concept  that  as  the  network  centricity  of  the  force 
increases.  Mission  Capability  Packages  (MCPs)  will  co-evolve.  MCP  elements,  such  as 
command  arrangements,  doctrine,  training,  etc.,  will  be  modified  as  the  transition  to  an 
Information  Age  military  organization  progresses.  The  NCO  CF  provides  a  means  to  measure  the 
extent  of  the  co-evolution  of  MCP  elements. 

2.1.2  Structure  of  the  Conceptual  Framework 

Each  concept  in  the  top-level  is  described  by  a  set  of  attributes  and  metrics  at  the  second  level. 
The  attributes  measure  characteristics  of  the  concept  in  terms  of  quantity  (how  much?  how 
often?  how  long?  etc.)  and  quality  (how  correct?  how  appropriate?  how  complete?  etc.).  Each 
attribute  is  actually  measured  by  a  metric  (or  set  of  metrics)  that  specifies  in  detail  what  data 
would  be  needed  to  measure  the  attribute.  Eor  instance,  the  “Degree  of  Networking”  is 
comprised  of  net  ready  nodes  and  the  network.  In  order  to  assess  the  impact  of  various  levels  and 
qualities  of  networking  on  force  performance  and  outcomes,  it  is  necessary  to  measure  these 
levels  and  qualities.  Eor  example,  as  Eigure  2-2  illustrates,  the  attributes  of  net  ready  nodes  are: 
Capacity,  Connectivity,  Post  and  Retrieve  Capability  Support,  Collaboration  Support,  and  Node 
Assurance.  The  attributes  of  the  network  are:  Reach,  Quality  of  Service,  Network  Assurance,  and 
Network  Agility.  In  order  to  gather  data  to  assess  each  of  these  attributes,  specific  metrics  are 
needed.  The  Conceptual  Eramework  provides  metrics  for  each  attribute.  Eor  example.  Network 
Reach  can  be  measured  by  the  percentage  of  nodes  that  can  communicate  in  desired  access 
modes,  information  formats,  and  applications. 


^  Section  4.1.5  below  discusses  Agility  in  some  detail. 
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NCO  Conceptual  Framework 
Top  level  and  Second  Level 


Top  Level  Coneepts  Seeond  Level  Attributes 

-= - = — — — : - — I  and  Metries 


Eaeh  Coneept  in  the  Top  Level  is  Mapped  to  Seeond 
Level  Attributes  and  Metries 


Figure  2-2.  Top  Level  and  Second  Level  View 

The  framework  is  a  rieh  set  of  metries  that  ean  be  utilized  to  evaluate  the  impaet  of  various  levels 
and  qualities  of  important  NCO  eoneepts,  sueh  as  the  degree  of  networking,  on  individual  and 
shared  information,  situational  awareness,  understanding,  deeisionmaking,  synehronization  of 
aetions  and  ultimately  effeetiveness.  The  NCO  Coneeptual  Framework  ean  be  utilized  in  a 
variety  of  ways.  For  instanee,  it  ean  be  used  as  a  tool  to  evaluate  foree  performanee  in  exereises 
and  experiments;  it  ean  also  be  used  to  guide  poliey  development  and  aequisition  deeisions.  In 
order  to  evaluate  the  relationships  among  the  eoneepts,  it  is  neeessary  to  establish  speeifie 
hypotheses  that  link  the  top-level  eoneepts  and  seeond-level  attributes.  Figure  2-3  illustrates  this 
step. 
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Exogenous 
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Metrics  for 
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Figure  2-3.  Relationships  among  Concepts 

2.1.3  Application  of  the  NCO  Conceptual  Framework 

In  order  to  validate  and  refine  the  conceptual  framework,  it  must  be  applied  to  a  broad  range  of 
mission  areas  across  the  range  of  possible  military  operations.  It  should  be  vetted  across  the  DoD, 
allied,  and  coalition  military  partners,  as  well  as  other  Government  agencies  so  that 
improvements  and  refinements  can  be  made.  The  more  mature  NCO  CF  can  then  be  used  as  an 
assessment  tool  and  metrics  guide  to  inform  experimentation,  acquisition,  and  other 
Transformation  related  activities.  These  activities  form  the  core  of  the  Network  Centric 
Operations  Conceptual  Framework  Program,  a  current  initiative  of  the  Office  of  Force 
Transformation. 

2. 1. 4  Air-to-Air  Example 

In  an  initial  test  of  concept,  the  NCO  Conceptual  Framework  was  used  to  evaluate  the  results  of 
air-to-air  training  sorties  in  a  major  training  exercise.  Over  12,000  training  sorties  were 
conducted  using  two  distinct  information  systems:  voice  only  and  voice  plus  link- 16.'°  The 
voice  plus  link- 16  system  is  illustrative  of  a  “networked”  force  in  that  all  force  members  shared 
voice  and  data  over  the  network.  The  voice  only  system  allowed  for  information  sharing  via 
voice  links  only  and  had  no  data-sharing  capabilities.  The  kill  ratio  was  over  two  and  a  half  times 
higher  for  the  “networked”  system  vs.  voice  only. 


***  JTIDS  Operational  Special  Project  (OSP)  Report  to  Congress,  Mission  Area  Director  for  Information  Dominance, 
Office  of  the  Secretary  of  the  Air  Force  for  Acquisition.  Washington,  DC:  Headquarters  U.S.  Air  Foree,  Deeember 


1997. 
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The  RAND  research  team  developed  an  Analytica  model  to  estimate  values  for  selected  NCO 
metrics  and  aggregate  them  into  the  values  of  the  top-level  concepts.  As  Figure  2-3  illustrates, 
while  both  systems  started  with  the  same  “quality  of  organic  information”  the  degree  of 
networking,  quality  of  shared  information,  awareness,  understanding,  decision  making  and 
effectiveness  diverged  significantly  between  the  two  systems. 


Degree  of 
Information 
“Share-ability’i.o 


Quality  of 
Networking 


Quality  of 
Organic  Info 


Quality  of  Individual 
Information 

.91 


Degree  of  Shared 
Information 


Voice  A 
Voice  +  Link  16 


Degree  of 
Shared 
Sensemaking 


Degree  of 
Actions/  Entities 
Synchronized 


S.ll:l 

Kill  Ratio 
(Effectiveness) 

Overall  average  over  information  quality 
dimensions  and  package  members 


Figure  2-3.  Comparison  of  MCPs  across  Voice  and  Voice  Plus  Link  16  Systems 

2.1. 5  Limits  of  the  NCO  Conceptual  Framework 

It  is  important  to  explicitly  identify  the  limitations  of  the  NCO  CF.  First  and  foremost,  the  NCO 
CF  is  a  work  in  progress.  This  document  is  the  initial  articulation  of  the  theory  and  thinking 
behind  the  CF.  Future  versions  are  expected.  Second,  the  NCO  CF  is  best  thought  of  as  an 
assessment  tool  that,  in  order  to  be  useful,  must  be  applied  in  a  specific  context.  As  the 
Conceptual  Framework  is  used  in  experiments,  investments  analyses,  and  applied  to  case  studies, 
evidence  will  be  accumulated  that  will  enable  us  to  specify  the  conditions  under  which  NCO 
hypotheses  are  supported.  Until  that  time,  however,  the  Conceptual  Framework  is  “neutral”  in 
that  it  is  not  a  prescriptive  guide,  that  is,  it  does  not  tell  us  “what  to  do”  in  order  to  become  more 
net-centric.  It  does  not  tell  us  “how  much”  is  enough  in  terms  of  network-centric  technologies 
and  practices.  The  Conceptual  Framework  will,  however,  facilitate  the  collection  of  the  evidence 
needed  to  answer  such  questions. 
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3.0  NCO  Conceptual  Framework:  Structure 

3.1  Overview 

While  the  NCO  Coneeptual  Framework  should  be  seen  as  mechanism  to  assess  the  structure  and 
processes  inherent  in  command  of  future  military  forces  and  control  of  those  same  forces  in  an 
operating  environment,  it  can  also  be  understood  as  a  set  of  tightly  coupled  concepts  and 
relations.  The  top  level  view  (Figure  2-1)  has  been  developed  to  show  those  factors  considered 
most  important.  However,  NCO  is  a  rich  and  evolving  set  of  ideas,  so  some  significant  features 
have  necessarily  been  captured  in  the  second  layer  (attributes  and  metrics).  Moreover,  because 
the  Framework  remains  a  work  in  progress,  all  the  top  level  concepts  and  the  relationships  shown 
between  them  (either  by  arrows  or  by  “nesting”  some  concepts  inside  others)  must  be  understood 
as  hypotheses  subject  to  disconfirmation  or  improvement  (by  improved  definition,  discovery  of 
limiting  conditions,  merging  some  ideas  and  distinguishing  others,  or  by  “discovering”  [deciding 
as  a  community]  that  other  concepts  must  be  included). 

3.2  Domains 

In  order  to  understand  Network  Centric  Operations,  it  is  essential  to  recognize  that  military 
entities  and  activities  are  located  in  four  domains:  the  physical,  information,  cognitive,  and  social 
domains.  The  first  three  domains  are  discussed  in  detail  in  Understanding  Information  Age 
Warfare.  The  physical  is  where  strike,  protect,  and  maneuver  take  place  across  the 
environments  of  sea,  air,  and  space.  The  information  domain  is  where  information  is  created, 
manipulated,  value-added  and  shared.  It  can  be  considered  the  “cyberspace”  of  military 
operations.  The  cognitive  domain  is  where  the  perceptions,  awareness,  understanding,  decisions, 
beliefs,  and  values  of  the  participants  are  located.  These  intangibles  are  crucial  elements  of 
network  centric  operations. 

The  social  domain  is  an  innovation  of  the  NCO  Conceptual  Framework.  It  is  where  force  entities 
interact,  exchanging  information,  awareness,  understandings  and  making  collaborative  decisions. 
It  overlaps  with  the  information  and  cognitive  domain  but  is  distinct  from  both.  Cognitive 
activities  by  their  nature  are  individualistic;  they  occur  with  in  the  minds  of  individuals. 
However,  shared  sensemaking,  the  process  of  going  from  shared  awareness  to  shared 
understanding  to  collaborative  decisionmaking,  can  be  considered  a  socio-cognitive  activity  in 
that  individual’s  cognitive  activities  are  directly  impacted  by  the  social  nature  of  the  exchange 
and  vice  versa. 

The  social  domain,  as  a  recent  innovation  to  network-centric  theory,  will  require  additional 
research  and  thinking.  Section  4.2  below  discusses  shared  sensemaking  in  some  detail  and 
provides  additional  insight  into  the  social  domain. 

These  four  domains  are  represented  in  the  NCO  Conceptual  Framework  by  the  color  scheme 
illustrated  in  Figure  3-1. 


*’  Understanding  Information  Age  Warfare:  pg  10-14. 
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Physical  Domain 

where  strike,  proteet,  and  maneuver  take  plaee  aeross 
different  environments 


Information  Domain 

where  information  is  created,  manipulated  and  shared 

Cognitive  Domain 

where  perceptions,  awareness,  beliefs,  and  values  reside 
and  where,  as  a  result  of  sensemaking,  decisions  are  made 

Social  Domain 

interactions  between  and  among  force  entities 


Figure  3-1.  The  Four  Domains 


3.3  The  Force 

The  Force  is  not  in  any  one  of  the  four  key  dimensions  (physical,  information,  cognitive,  or 
social)  because  it  is  both  in  them  all  and  also  contains  elements  of  all  of  them.  The  Force,  in  this 
framework,  is  made  up  of  entities  that  provide  four  basic  kinds  of  functions  (see  Figure  3-2): 

■  Effects  -  those  elements  of  the  force  that  play  a  role  by  impacting  the  operating 
environment.  These  certainly  include  weapons,  but  they  also  include  information  entities 
such  as  psychological  operators  and  media  spokespersons,  electronic  and  cyberspace 
warfare  entities,  and  those  responsible  for  diplomatic,  economic,  and  other  effects. 

■  Information  Sources  -  force  elements  that  collect  or  generate  information  relevant  to 
operations.  These  include  sensors  of  all  types,  human  intelligence,  and  organizations  that 
collect  relevant  information  from  open  sources  (news  reports,  web  sites,  etc.). 

■  Value  added  services  -  those  elements  that  fuse  data  or  information,  add  knowledge  to 
help  understand  or  interpret  data,  information  or  other  knowledge,  respond  to  queries,  or 
define  needs  for  new  data  or  information. 

■  Command  and  control  -  those  force  elements  that  perform  or  support  the  command 
function  or  the  control  function,  including  decision  support  tools. 
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Figure  3-2.  Relationship  of  MCPs  to  the  Force 


A  single  platform,  work  station,  or  headquarters  may  perform  more  than  one  of  the  four  roles  in 
the  foree.  An  aireraft  earner,  for  example,  will  typically  include  sensors,  value-added  serviees, 
C2,  and  effeetors  in  the  same  operation.  With  the  ehanging  nature  of  the  strategie  environment, 
U.S.  foree  entities  will  inereasingly  be  required  to  be  effective  across  a  broader  range  of 
functions.  This  will  require  agility. 


3.4  NCO  Value  Chain 


The  tenets  of  NCW  state  that  a  robustly  networked  foree  will  ultimately  result  in  dramatieally 
improved  mission  effectiveness.  The  intervening  steps  that  lay  between  a  ehange  in  the  network 
eentric  eapabilities  of  the  foree  and  mission  outcomes  can  be  eonsidered  the  NCO  “Value 
Chain.”  The  value  chain  actually  is  a  way  to  represent  key  hypotheses  of  network  centric  theory. 
An  important  goal  of  the  NCO  Conceptual  Framework  is  to  allow  researehers  to  colleet  evidenee 
(using  a  set  of  metrics)  on  these  hypotheses  so  that  they  ean  be  evaluated  scientifically.  The 
Conceptual  Framework,  then,  ean  be  considered  an  elaborated  NCO  Value  Chain.  Figure  3-3 
illustrates  how  the  NCO  CF  ean  represent  the  NCO  Value  Chain  in  the  RAND  Air-to-Air 
example. 
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Figure  3-3.  The  NCO  Value  Chain  Storyline 


3.5  Top  Level  Concepts 

The  NCO  Conceptual  Framework  is  comprised  of  the  following  top-level  concepts: 

■  Quality  of  Organic  Information 

■  Quality  of  Networking 

■  Degree  of  Information  “Shareability” 

■  Quality  of  Individual  Information 

■  Quality  of  Individual  Sensemaking 

■  Quality  of  Interactions 

■  Degree  of  Shared  Information 

■  Degree  of  Shared  Sensemaking 

■  Degree  of  Decision/Synchronization 

■  Degree  of  Actions/Entities  Synchronized 

■  Degree  of  Effectiveness 

■  C2  Agility 

■  Eorce  Agility 


This  compares  to  the  seven  concepts  specified  in  the  original  Tenets: 


•  Robustly  networked  force 

•  Information  sharing 

•  Collaboration 

•  Quality  of  information 

•  Shared  situational  awareness 

•  Self-synchronization 

•  Sustainability  and  speed  of  command 
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There  is  an  obvious  trade  off  between  simplieity  and  eomplexity  in  the  representation  of  the 
NCO  Value  Chain.  For  some  applieations,  sueh  as  those  for  senior  level  deeision  makers  or 
broad  non-DoD  audienees  interested  in  learning  about  NCO,  a  simplified  presentation  sueh  as 
the  original  tenets  is  preferred.  For  applieations,  such  as  case  studies,  experiments,  or  specific 
acquisition  decisions,  a  more  detailed  and  complex  representation  is  required.  The  NCO 
Conceptual  Framework,  in  its  current  state,  is  intended  for  the  latter  audience.  It  is  meant  to 
provide  guidance  to  researchers  and  decision  makers  conducting  experiments,  case  studies,  and 
making  specific  acquisition  decisions  and  as  such  in  necessarily  complex. 


3.6  Attributes  and  Metrics 

In  order  to  assess  the  impact  of  network  centric  technologies  on  each  of  the  top-level  concepts  in 
the  NCO  Value  Chain,  it  is  necessary  to  use  a  set  of  attributes  that  allow  us  to  assess  the  different 
characteristics  each  top-level  concept.  Likewise,  each  attribute  can  be  measured  with  a  specific 
metric,  or  set  of  metrics. 

Assessing  the  “dependent  variable,”  mission  effectiveness,  requires  that  we  consider  that 
particular  concept  in  the  context  of  a  set  of  measures  of  effectiveness.  It  is  useful  to  think  in 
terms  of  a  hierarchy  of  measures  that  allow  one  to  measure  performance  across  different  levels 
of  the  relevant  outcome  space.  The  following  hierarchy  of  measures  is  adapted  from  the  NATO 
Code  of  Best  Practices  for  C2  Assessment: 

■  Measures  of  Policy  Effectiveness  (MoPE),  which  focus  on  high-level  policy  outcomes; 

■  Measures  of  Eorce  Effectiveness  (MoEE),  which  focus  on  the  extent  to  which  a  force 
achieves  its  mission/objectives; 

■  Measures  of  C2  Effectiveness  (MoCE),  which  focus  on  the  impact  of  C2  on  outcomes; 
and 

■  Measures  of  C2  Performance  (MoCP),  which  focus  on  internal  characteristics  of  C2 
programs,  initiatives,  system(s),  etc. 

In  determining  what  attributes  are  appropriate,  it  is  necessary  to  begin  with  the  concepts  of 
interest  and  their  definitions.  Attributes  must  provide  a  means  of  measuring  the  actual  concept  as 
specified  in  the  definition.  Similarly,  metrics  must  be  selected  that  actually  allow  one  to  measure 
the  concept  of  interest.  Otherwise,  our  attributes  and  metrics  would  not  be  valid  indicators  of  the 
values  of  the  concepts.  In  addition,  metrics  should  be  reliable,  that  is,  they  are  specified  with  the 
appropriate  level  of  clarity  and  detail  that  multiple  attempts  (by  different  researchers)  to  measure 
an  attribute  using  a  given  metric  will  result  in  the  same  value  for  the  attribute. 

It  is  helpful  to  identify  broad  classes  of  attributes  that  can  be  utilized  in  such  an  effort.  These 
include;  objective  attributes,  fitness-for-use  attributes,  agility  attributes,  and  other  concept 
specific  attributes.  These  are  defined  below. 


Stenbit,  John  P.,  Linton  Wells,  II.,  &  David  S.  Alberts.  NATO  Code  of  Best  Practice  for  C2  Assessment. 
Washington,  DC:  CCRP  Publieation  Series.  2002:  pg  92. 
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Objective  Attributes  measure  quality  in  referenee  to  eriteria  that  are  independent  of  the 
situation.  For  example,  the  currency  of  a  given  data  element  indieates  the  age  of  the  information 
available  and  ean  be  expressed  in  units  like  minutes,  hours,  days,  etc. 

Fitness-for-Use  Attributes  measure  quality  in  reference  to  criteria  that  are  determined  by  the 
situation.  For  example,  the  timeliness  of  a  given  data  element  indicates  the  extent  to  which  the 
information  is  received  in  a  time  that  is  appropriate  for  its  intended  use.  What  is  appropriate  is 
context  dependent.  In  some  contexts  a  currency  of  two  hours  is  adequate,  where  as  in  other 
contexts  a  currency  of  two  minutes  is  what  is  needed.  Fitness-for-use  attributes  allows  one  to 
capture  information  that  is  context  dependent. 

Agility  Attributes  measure  the  aspects  of  agility  across  the  six  dimensions.  These  attributes 
inherently  are  comparative,  i.e.  agility  implies  an  ability  to  change  over  time  and,  as  such,  the 
values  of  the  metrics  for  these  attributes  have  to  be  compared  to  some  baseline  values. 

Concept  Specific  Attributes  measure  unique  aspects  of  some  concepts.  For  instance, 
synchronicity  is  an  attribute  of  the  Quality  of  Interactions  concept  that  measures  the  extent  to 
which  C2  processes  are  effective  across  time  (synchronous  vs.  asynchronous)  and  space  (co¬ 
located  vs.  distributed).  This  attribute  is  appropriate  in  determining  the  extent  to  which  elements 
in  a  C2  organization  can  interact  simultaneously  in  time  and  space  but  is  not  necessarily  relevant 
to  other  concepts. 

Attributes  will  likely  be  measured  by  metrics  that  are  subjective  (qualitative)  as  well  as  those  that 
are  objective  (quantitative).  Whenever  possible,  quantitative  metrics  should  be  utilized,  and  the 
goal  is  to  increasingly  rely  on  quantitative  metrics.  However,  there  are  circumstances  when 
qualitative  metrics  are  appropriate  and  necessary.  Qualitative  metrics  should  be  built  on  clearly 
articulated  criteria  identified  by  subject  matter  experts,  or  determined  by  existing  theory  and/or 
empirical  observations. 

It  is  important  to  keep  in  mind  that  summary  attributes  (and  metrics),  aggregated  across 
programs,  initiatives,  systems,  etc.,  are  often  utilized  in  high-level  comparative  assessments.  In 
such  cases,  attention  to  standard  multi-attribute  measurement  methods  is  essential.  Most 
importantly,  the  level  of  precision  of  the  aggregate  metrics  cannot  exceed  the  level  of  precision 
of  the  least  precise  input  metric. 

Another  important  issue  to  consider  is  the  distinction  between  metrics  and  objectives.  Metrics  are 
the  yardsticks  that  we  use  to  measure  aspects  of  attributes.  Objectives  are  the 
performance/quality  goals.  For  instance,  the  attribute  completeness  is  the  extent  to  which  shared 
understanding  incorporates  all  relevant  information  and  possible  outcomes.  The  metric  could  be 
measured  in  percentages.  An  objective  may  be  80%  or  95%,  for  example.  The  metric  is  what  is 
being  measured;  the  objective  is  some  goal  that  must  be  determined  by  policy,  experimentation 
and/or  analysis. 

Figure  3-4  illustrates  the  relationship  between  concepts,  attributes  and  metrics  for  the  concept 
Quality  of  Organic  Information. 
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Figure  3-4.  Quality  of  Organic  Information 
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4.0  NCO  CF  Concepts,  Attributes,  and  Metrics  in  Detail 

This  section  provides  detailed  descriptions  of  each  of  the  major  concepts  in  the  top-level  of  the 
NCO  Conceptual  Framework.  Following  the  guidance  of  the  NATO  Code  of  Best  Practice  for  C2 
Assessment,  it  starts  “at  the  bottom”  of  the  NCO  Conceptual  Framework  with  a  description  of  the 
“dependent”  variables  in  the  NCO  Value  Chain:  Synchronization,  Agility,  and  Effectiveness.  It 
then  discusses  the  concepts  that  trace  out  the  value  chain  from  networking  and  information, 
through  Sensemaking,  and  Quality  of  Interactions.  This  section  concludes  with  a  summary  and 
synthesis  of  the  approach  taken  to  relate  concepts,  attributes  and  metrics. 

4.1  Synchronization,  Agility,  and  Effectiveness 


NCO  Conceptual  Framework 


Figure  4-1.  The  NCO  Conceptual  Framework 


4.1.1  Decision  Synchronization 

Stated  most  simply,  decisions  are  choices  among  alternatives.  In  the  context  of  the  NCO 
Conceptual  Framework,  they  may  take  place  across  multiple  levels  of  command:  from  command 
intent  generated  at  the  most  senior  level  and  disseminated  to  all  levels  and  across  all  functions  to 
selection  of  courses  of  action  across  echelons,  functions,  organizations  (including  coalition 
partners  and  non-military  organizations  involved  in  missions  with  the  military,  particularly  when 
effects-based  operations  are  explicitly  considered),  to  tactical  decisions  ranging  from  weapons- 
target  pairing  to  when  contingencies  should  be  activated. 

Many  decisions,  particularly  in  Industrial  Age  militaries,  or  the  Industrial  Age  legacy  parts  of 
Information  Age  militaries,  are  expressed  as  plans.  Military  plans  may  be  more  or  less  detailed, 
but  they  always  include  or  imply  six  elements: 

■  Missions  -  what  is  to  be  accomplished; 
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■  Assets  -  which  resources  (including  elements  of  the  force)  are  assigned  or  available  for 
each  mission  or  military  task; 

■  Command  Arrangements  -  what  are  the  organizational  relationships  among  the  elements 
of  the  force  and  between  the  force  and  other  organizations  that  the  force  depends  upon; 

■  Boundaries  -  which  organizations  are  responsible  for  and  have  authority  over  what 
geographic  regions  (land,  air,  maritime,  and  space)  and  functions; 

■  Schedules  -  how  are  missions  and  assets  organized  over  time;  and 

■  Contingencies  -  explicitly  recognizable  situations  under  which  missions,  assets, 
command  arrangements,  boundaries,  or  schedules  will  change. 

Explicit,  written  plans  are  not  essential  in  all  military  operations.  In  many  dynamic  situations, 
particularly  in  Information  Age  militaries  with  very  flat  organizational  structures  and  doctrines 
that  encourage  self-synchronization,  plans  may  be  largely  implicit,  expressed  very  briefly,  and 
depend  on  prior  training  and  shared  mental  models.  For  example,  in  the  NCO  Conceptual 
Framework  Air-to-Air  combat  case  study  RAND  conducted  on  the  effect  of  Fink  16  on  combat 
power,  pilots  were  able  to  synchronize  their  tactical  actions  with  a  minimum  of  discussion 
because  of  their  prior  training  and  shared  command  intent  allowed  the  participants  to  effectively 
and  efficiently  integrate  the  shared  information  provided  by  the  Fink  16  system  to  form  shared 
understanding  of  the  battles  space. 

4.1.2  Action/Entity  Synchronization 

Synchronization,  the  third  key  element  in  this  conceptual  area,  is  defined  as  “purposeful 
arrangement  in  time  and  space.”  While  a  rich  concept,  its  meaning  in  the  context  of  future 
command  and  control  has  thus  far  been  confined  to  a  single  dimension  with  three  defined  scale 
points  (see  Figure  4-2): 


Conflicted  Deconflicted  Synergistic 


Figure  4-2.  Synchronization  Categories 


■  Conflicted  -  two  actions  or  entities  interfere  with  one  another.  The  classic  case  of 
conflicted  actions  is  casualties  due  to  friendly  fire.  Another  good  example  is  traffic  jams 
when  logistics  trains  from  two  forces  block  one  another’s  routes. 

■  Deconflicted  -  actions  or  entities  that  are  prevented  from  interfering  with  one  another  by 
separation  in  time,  space,  or  both.  Most  classic  military  control  measures  from  the 
Industrial  Age  are  deconfiiction  tools.  Ground  units  are  given  specific  areas  of 
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responsibility,  no-fire  lines  are  established  to  prevent  fratricide,  roads  are  reserved  for  the 
exclusive  use  of  particular  organizations,  fixed  wing  aircraft  and  rotary  wing  aircraft  are 
assigned  different  altitudes,  etc. 

■  Synergistic  -  actions  and  entities  that  reinforce  one  another’s  desirable  impacts  on  the 
operating  environment.  For  example,  the  ability  to  strike  suddenly  and  fiercely  without 
warning  (precision  guided  munitions  delivered  at  night  from  high  altitudes  or  by  stealth 
aircraft)  may  be  synergistic  with  carefully  designed  leaflets  because  together  they  yield 
larger  numbers  of  surrenders  and  desertions  than  either  of  them  would  produce  alone. 
More  traditionally,  combined  arms  teams  can  accomplish  missions  that  no  single  arm 
(infantry,  armor,  artillery,  and  aviation)  can  achieve  independently. 

4. 1. 3  Measuring  Synchronization  of  Decisions  and  Plans 

There  are  four  dimensions  across  which  the  synchronization  of  decisions  and  plans  can  be 
measured  -  entities,  expected  actions,  plan  elements,  and  time.  No  single  attribute  is  a  perfect 
measure  for  synchronization  and  they  may  well  interact.  In  particular,  time  may  be  a  control 
factor  for  entities,  actions,  and  plan  elements.  The  four  attributes  are  really  just  four  different 
perspectives  on  synchronization  during  the  process  of  integrating  decisions  or  creating  plans. 

The  crucial  decision  an  analytic  team  must  make  is  the  level  of  detail  at  which  measurement  will 
be  made.  The  simplest  case  is  that  of  entities.  In  any  battlespace  or  operating  environment  there 
are  a  number  of  different  levels  at  which  entities  can  be  counted,  including  not  only  echelon,  but 
also  function.  So,  decisions  or  the  plans  that  embody  them  can  be  counted  at  any  of  those  levels. 
However,  selecting  a  level  then  implies  the  ability  to  operationalize  that  level  as  well  as  count 
consistently.  If,  for  example,  the  entities  selected  are  combat  flights  of  aircraft,  then  all  combat 
flights  must  be  counted,  and  only  those  related  to  combat  flights,  not  flights  for  other  purposes.  If 
the  level  selected  for  a  ground  combat  effort  is  the  company,  then  its  equivalents  must  be 
understood  in  terms  of  other  types  of  entities  (for  example,  artillery  batteries).  Deciding 
equivalence  will  sometimes  be  difficult  if  the  entities  involved  are  heterogeneous.  For  example, 
naval  forces  are  organized  very  differently  from  air  and  ground  forces,  so  deciding  the  level  of 
analysis  for  an  amphibious  operation  may  require  considerable  thought.  Identifying  peers  in 
functional  relationships  is  probably  a  good  way  to  organize  this  problem. 

Equivalent  decisions  must  also  be  made  for  plan  elements,  actions,  and  the  timesteps  to  be  used 
in  measurement.  Plan  elements  (missions,  assets,  command  arrangements,  boundaries,  schedules 
and  contingencies)  are  often  controlled  by  considering  the  number  of  entities  they  include,  so 
mission  assignment  is  the  number  of  missions  times  the  number  of  entities  tasked  to  conduct  and 
support  that  mission.  Similarly,  expected  actions  can  be  controlled  for  the  number  of  entities 
anticipated  to  participate  in  or  support  them. 

4. 1. 4  Relevant  Metrics 

Looked  at  through  these  lenses,  the  metrics  for  decision  and  plan  synchronization  will  be 
computed  as: 

■  The  percentage  of  entities  included  in  decisions  that  are  conflicted,  deconfiicted,  or 
synergistic; 
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■  The  percentage  of  plan  elements  that  are  conflicted,  deconflicted,  or  synergistic; 

■  The  percentage  of  expected  actions  that  are  conflicted,  deconflicted,  or  synergistic;  and 

■  The  percentage  of  time  that  decisions  and  plans  (seen  as  two  representations  of  the  same 
thing)  are  conflicted,  deconflicted,  or  synergistic. 

Note  that  conflicted  entities,  plan  elements,  expected  actions,  and  periods  of  time  are  always 
negative.  They  literally  represent  the  fog  and  friction  of  war  in  this  portion  of  the  NCO 
Conceptual  Framework.  Similarly,  synergistic  entities,  plan  elements,  expected  actions,  and 
periods  of  time  are  always  good.  However,  note  that  some  synergy  can  be  better  than  other 
synergy  because  the  value  of  synergy  is  not  absolute,  but  is  derived  from  its  impact  on 
effectiveness.  Finally,  deconfliction  is  always  better  than  being  conflicted  and,  by  definition,  less 
valuable  than  synergy.  However  these  differences  are  ordinal  (there  is  no  way  to  measure  the 
difference  between  the  three  values).  In  theory,  deconfliction  may  be  barely  better  than  being 
conflicted  or  just  short  of  being  synergistic. 

Future  challenges  in  this  area  include:  (a)  establishing  the  relationships  between  synergy  in 
decisions  and  plans  in  the  cognitive  domain  and  synchronization  of  actions  and  entities  in  the 
physical  domain  as  well  as  with  effectiveness,  and  (b)  developing  more  precise  ways  of 
measuring  synchronization.  The  three-part  scale  used  now  (conflicted,  deconflicted,  and 
synergistic)  appears  valid  and  reliable,  but  has  limited  precision.  The  existing  scale  also  appears 
to  have  credibility  with  military  professionals  and  analysts  in  large  measure  because  it  is 
transparent  (easily  understood  and  expressed  in  examples).  Better  measurement  approaches  will 
need  to  preserve  the  validity,  reliability,  and  credibility  of  the  current  system  while  providing 
greater  precision.  Case  studies  and  other  research  (particularly  properly  focused  simulations) 
should  help  to  establish  the  correlation  (or  lack  of  correlation)  and  conditions  influencing  the 
relationship  between  synchronization  in  the  cognitive  domain  and  both  effectiveness  and 
synchronization  in  the  physical  domain. 

4.1. 5  Degree  of  Action  and  Entity  Synchronization 

Moving  to  the  physical  domain,  the  concept  of  synchronization  remains  unchanged  -  purposeful 
arrangement  in  time  and  space.  There  are  three  relevant  dimensions  -  entities,  actions,  and  time. 
The  same  fundamental  analytic  problems  remain,  especially  the  need  to  identify  and  apply  a 
level  of  analysis  consistently  and  to  make  time  a  meaningful  control.  The  implied  metrics  are: 

■  The  percentage  of  entities  that  are  categorized  as  conflicted,  deconflicted,  and 
synergistic; 

■  The  percentage  of  actions  categorized  as  conflicted,  deconflicted,  and  synergistic;  and 

■  The  percentage  of  time  that  the  force  is  classified  as  conflicted,  deconflicted,  and 
synergistic. 

This  last  metric  may  be  very  difficult  to  operationalize  outside  the  context  of  entities  or  actions, 
so  time  may  be  best  considered  as  a  control,  making  the  metrics  of  interest  the  rate  of  conflicted 
actions  or  entities  over  units  of  time.  This  may  well  be  the  best  way  to  approach  synchronization 
in  a  dynamic  environment  where  entities  may  change  their  degree  of  synchronization  over  time 
(move  in  and  out  of  the  three  categories)  and  actions  will  have  a  temporal  focus.  Of  course,  the 
“snapshot”  approach  may  also  be  useful.  This  technique  would  call  for  assessing  the  state  of 
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synchronization  at  selected  points  in  time,  whieh  might  be  identified  beeause  of  their  substantive 
importance  (just  prior  to  hostilities,  two  hours  after  a  major  attack  by  the  red  forces,  etc.),  at 
systematic  points  in  time  (every  four  hours),  or  on  the  basis  of  a  stratified  sample. 

4. 1. 6  Degree  of  Effectiveness 

Effectiveness  always  deals  with  impact  on  the  operating  environment.  The  MORS  (Military 
Operations  Research  Society)  work  on  measures  of  merit  during  the  early  1980s  has  been  widely 
accepted  in  the  C4ISR  analytic  community.  It  has  been  most  recently  integrated  in  the  NATO 
Code  of  Best  Practice  for  C2  Assessment  and  has  been  used  by  JFCOM  and  others  in  significant 
experiments.  It  reeognizes  three  levels  of  Measures  of  Effectiveness  (MoE),  Measures  of  C2 
Effectiveness  (MoCE),  Measures  of  Force  Effectiveness  (MoFE),  and  Measures  of  Policy 
Effectiveness  (MoPE).  This  last  eategory  was  added  by  a  NATO  Studies  and  Analysis  panel 
working  group  (SAS-026)  in  recognition  of  the  fact  that  military  performance  does  not  guarantee 
accomplishing  the  larger  missions  implied  in  effects-based  operations  (peacekeeping,  nation 
building,  etc.).  That  working  group  included  El.S.  members  who  had  helped  develop  the  original 
MORS  taxonomy. 

The  three  categories  can  be  distinguished  by  an  example.  The  context  is  a  carrier  battle  group 
positioned  off  the  shore  of  a  friendly  state  (The  Republic  of  Goodness)  that  has  been  invaded  by 
a  neighbor  (The  Kingdom  of  Badness).  Badness  posses  medium  range  bombers  equipped  with 
cruise  missiles.  The  ability  of  the  C2  system  supporting  the  battle  group  to  identify  potential 
threat  platforms  early  enough  to  position  defensive  platforms  (aircraft  and  vessels)  where  they 
can  intercept  and  engage  them  beyond  the  stand  off  range  of  the  cruise  missiles  is  an  indieator  of 
C2  effectiveness  (MoCE).  This  might  be  measured  by  how  rapidly  threat  platforms  can  be 
identified  and  plans  developed  and  implemented  to  intercept  them  and  how  correct  is  the 
identification,  and  so  forth. 

However,  the  C2  system  cannot  engage  platforms  in  the  physieal  domain.  That  is  accomplished 
by  effectors.  The  correct  MoFE  (measures  of  force  effectiveness)  involve  adversary  kills, 
mission  aborts  by  platforms  that  launch  beyond  effective  range  or  break  off  before  launching, 
and  casualty  ratios.  The  “kill  ratios”  employed  by  RAND  in  the  initial  air-to-air  case  study  in 
support  of  the  NCO  Conceptual  Framework  are  MoFE.  In  the  event  that  the  efforts  to  protect  the 
carrier  battle  group  result  in  shooting  down  innocent  aircraft  by  accident  and  therefore  reducing 
support  for  the  coalition  and  endangering  the  mission,  then  MoPE  (Measures  of  Policy 
Effectiveness)  have  been  impacted. 

Most  of  the  MoCE,  in  applying  the  NCO  Conceptual  Framework,  will  occur  in  the  sensemaking 
conceptual  arena.  However,  analytic  teams  should  also  be  alert  to  the  possibility  that  the  C2 
systems  (human  and  machine)  involved  in  a  particular  operation,  exercise,  or  simulation  may 
cause  physical  movement  of  force  elements.  This  will  appear  in  the  synchronization  metrics  and 
signal  that  MoFE  should  be  applied. 

Effectiveness  metrics  share  with  synehronization  metrics  the  need  to  identify  an  appropriate  level 
of  analysis.  When  applying  the  NCO  Coneeptual  Framework  at  the  tactical  level,  that  is  when  it 
is  used  to  evaluate  specific  case  studies  or  utilized  in  specific  experimentation  efforts,  the  key 
units  are  clearly  missions.  Missions  may  be  combat  missions,  combat  support  missions  or 
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missions  in  which  the  military  supports  others  (humanitarian  missions,  support  to  law 
enforeement,  non-eombat  evaeuations).  However,  there  will  often  be  layers  of  missions  assigned 
to  different  elements  of  the  foree  (entities),  in  different  funetional  areas  (logisties,  intelligence, 
ete.),  and  over  time.  Henee  the  degree  of  mission  aeeomplishment  may  differ  aeross  these  arenas 
and  the  relevant  metries  will  include  both  assigning  values  to  individual  metries  and  “roll  up” 
ealeulations  that  ereate  mission  aeeomplishment  indiees.  As  with  synehronization  metries,  time 
or  periods  of  time  may  need  to  be  eonsidered.  In  past  analyses,  sampling  has  been  used  as  has 
assessment  based  on  phases  of  the  mission  (pre-deployment,  deployment,  movement  to  eontaet, 
eombat  by  phases,  eonfliet  termination).  However,  not  all  military  missions  will  be  organized  in 
this  way. 

Some  missions  have  relatively  straightforward  or  direet  measures.  For  example,  the  RAND  air 
combat  case  study  simply  used  loss  ratios  to  assess  the  effeetiveness  of  the  entities.  However, 
analysts  should  be  eautious  when  developing  or  seleeting  MoFE  beeause  they  may  miss 
important  distinetions.  For  example,  when  assessing  sueeess  in  interdieting  drug  movements 
“suecesses”  include  not  only  eaptured  or  destroyed  loads,  but  also  aborted  missions  that  foreed 
the  smugglers  to  return  with  their  load.  These  aborted  missions  are  not  fully  sueeessful  in  that 
they  do  not  result  in  removing  the  drugs  from  the  pipeline.  However,  they  do  mean  that 
smugglers  must  try  again  to  eomplete  the  movement  (providing  another  opportunity  at  fully 
sueeessful  interdietion)  and  often  yield  intelligenee  about  where  the  load  was  diverted, 
identifying  smuggling  bases  or  transit  points.  Indeed,  aborted  missions  may  also  create  pressure 
within  the  smuggling  organizations  that  lead  them  to  take  ehanees,  open  up  new  routes,  or 
attempt  new  means  of  smuggling,  all  improving  the  ehanees  of  sueeessful  interdietion.  Henee, 
ignoring  the  “abort”  oases  would  give  a  false  pioture  of  the  impaot  of  interdiction  efforts. 

Similarly,  analysts  must  be  aware  of  MoPE  when  designing  or  seleeting  the  ways  they  will 
measure  effeetiveness.  Eor  example,  reoent  Army  experiments  with  Information  Age  oommand 
and  oontrol  oonoepts  introduoed  the  possibility  of  neutral  oasualties  into  their  soenarios  and  even 
kept  traok  of  them.  However,  the  analytio  team  ohose  to  measure  effeetiveness  only  including 
force  (red  and  blue)  oasualties.  As  an  unintended  oonsequence,  the  “players”  probably  did  not 
value  neutral  oasualties  as  muoh  as  they  might  have  in  a  “real”  eombat  situation.  At  a  minimum 
the  results  of  these  experiments  will  need  to  be  reanalyzed  to  make  useful  estimates  of  the  extent 
to  whioh  polioy  objeotives  (which  often  depend  on  attitudes  among  the  neutral  populations)  were 
being  oompromised  in  these  experiments. 

Mission  aeeomplishment  not  tied  to  physioal  objeotives  or  performanoe  oharaoteristios  (take  the 
hill  before  1400,  move  425  tons  of  artillery  ammunition  into  forward  supply  depots  before 
August  14)  oan  be  diffioult  to  assess  objeotively.  Where  direct  observation  is  possible,  sooring 
should  be  simple  (i.e.,  1  for  mission  aeeomplishment,  0  for  failure  to  aooomplish  the  mission, 
aggregated  over  missions  to  ereate  an  index).  Some  teehniques  that  have  been  employed 
sueeessfully  in  the  past  when  direet  measurement  is  not  possible  inelude: 

■  Independent  ratings  by  panels  of  subjeet  matter  experts;  and 

■  Convergenee  teehniques  using  group  software. 

Use  of  seleeted  Subjeet  Matter  Experts  (SMEs)  to  seore  effectiveness  has  been  used  in  eases 
where  a  modest  set  of  resources  were  available.  Eor  example,  in  a  multinational  limited  objeetive 
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experiment  organized  by  JFCOM  to  examine  hypotheses  about  alternative  methods  of  planning 
in  eoalitions,  five  flag  offieers  (aetive  duty  and  retired)  from  three  different  nations  were  used  to 
assess  the  quality  of  the  plans  produeed  using  different  proeedures  and  organizational  struetures. 
These  offieers  met  with  the  experimentation  team  to  discuss  the  scenario  being  used,  discussed 
alternative  approaches  (command  intent  and  courses  of  action)  among  themselves,  and  worked 
independently  to  develop  their  assessments.  Their  scoring  was  “blind.”  That  is,  they  did  not 
know  which  plan  had  been  developed  using  which  procedures  and  organizations.  The  results  of 
their  scoring  were  merged  using  a  voting  technique  (1  for  the  top  ranked  plan,  5  for  the  bottom) 
with  ties  allowed.  The  results  were  highly  consistent,  with  little  ambiguity  about  the  proper  rank 
order  of  the  plans.  However,  the  results  were  also  ordinal  (i.e.,  ranked)  so  that  it  was  impossible 
to  determine  the  size  of  the  differences  recorded  and  cannot  be  compared  with  other  scoring 
efforts  because  of  the  number  of  unique  factors  involved  in  the  scoring  (unique  SMEs,  unique 
scenario,  etc.). 

Finally,  increasing  use  is  being  made  in  a  number  of  communities  of  software  that  encourages 
discussion  and  voting  among  subject  matter  experts.  These  tools  use  a  combination  of 
discussion  and  voting  (anonymous  in  most  cases).  In  a  typical  example  the  discussion  leader 
introduces  the  topic  and  indicates  how  the  voting  will  be  organized  over  time  and  across  issues. 
Discussion  follows  in  which  the  structure  of  the  process,  the  meaning  of  key  terms,  and  the 
voting  procedure  are  typically  addressed.  Once  the  group  has  had  its  say  (and  appropriate 
adjustments  made),  voting  begins.  The  results  of  the  first  round  are  displayed  to  the  group  and 
discussion  encouraged,  particularly  on  the  part  of  those  whose  votes  are  outliers  and  those 
representing  major  positions.  Under  some  conventions,  no  one  is  required  to  speak.  Rounds  of 
votes,  display,  and  discussion  follow  until  convergence  occurs  or  alternative  positions  harden. 

Voting  and  display  techniques  are  attractive  when  working  with  genuinely  new  material  (for 
example  nontraditional  missions)  and  when  SMEs  from  very  different  communities  are  involved. 
They  can  also  be  useful  when  developing  MoE  in  complex  situations  with  the  expectations  that 
the  rules  and  structures  developed  will  be  reused  in  later  efforts.  However,  they  tend  to  develop 
relatively  unique  language,  processes,  and  voting  conventions  that  may  not  be  replicable  in  other 
contexts  or  with  other  groups,  so  they  should  be  used  with  some  care. 

4. 1. 7  Efficiency  and  Effectiveness 

Efficiency,  defined  in  classic  terms,  refers  to  the  cost  of  being  effective.  In  economics,  for 
example,  the  classic  measure  of  efficiency  is  units  of  output  per  unit  of  input  or  productivity.  A 
more  efficient  firm  or  process  generates  more  value  (typically  measured  in  dollars)  for  the  same 
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An  example  ean  be  found  in:  Addison,  Tom.  E-Commeree  Projeet  Development  Risks:  Evidenee  from  a  Delphi 
Survey.  International  Journal  of  Information  Management.  Volume  23,  Issue  1.  February  2003:  pg  25-40.  This 
paper  reports  on  a  study  to  determine  the  opinion  of  expert  praetitioners  of  the  most  important  risks  in  the 
development  of  e-eommeree  projeets.  See  also:  Madu,  Christian,  Chu-FIua  Kei,  Assumpta  Madu.  Setting  Priorities 
for  the  IT  Industry  in  Taiwan:  a  Delphi  Study.  Long  Range  Planning.  Volume  24,  Issue  5.  1991:  pg  17.  Taylor, 
Raymond,  David  Meinhardt.  Defining  Computer  Information  Needs  for  Small  Business:  A  Delphi  Method.  Journal 
of  Small  Business  Management.  Volume  23.  April  1985:  pg  3. 
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or  less  cost  of  input  (again  usually  expressed  in  dollars).  If  there  is  no  output,  there  is  no 
efficiency  because  there  can  be  no  relationship  between  units  of  output  and  units  of  input.  It  is 
like  trying  to  talk  about  the  efficiency  of  a  factory  that  cannot  produce  any  useful  products. 

In  military  terms,  the  outputs  of  value  are  accomplished  missions  -  effectiveness.  Hence,  the 
idea  of  efficiency  only  makes  sense  when  missions  are  accomplished.  When  that  occurs  it  makes 
sense  to  consider  efficiency  in  at  least  three  dimensions: 

■  the  size  of  the  force  required  to  accomplish  the  mission  (implicitly,  larger  forces  are  more 
costly  and  when  a  smaller  force  can  be  used  other  forces  are  available  to  take  on  other 
missions); 

■  the  casualties,  whether  measured  in  lives  and  treasure  are  smaller  (forces  that  accomplish 
missions  with  fewer  casualties  and  at  less  cost  to  their  national  treasuries  have  more 
capability  to  take  on  other,  subsequent  missions);  and 

■  the  time  required  to  accomplish  the  mission  (this  is  often  assumed  to  be  a  correlate  of 
fewer  casualties  and,  by  definition,  it  implies  that  the  same  force  or  force  elements  is 
available  sooner  for  other  missions). 

Indeed,  the  concept  of  NCO  is  attractive  not  only  because  it  is  hypothesized  to  increase  the 
likelihood  of  mission  accomplishment,  but  also  because  it  is  hypothesized  to  increase  the 
efficiency  of  the  force  when  compared  with  those  forces  that  are  not  network  centric.  Hence,  the 
NCO  Conceptual  Framework  ought  to  include  measures  of  efficiency  so  that  these  hypotheses 
can  be  examined. 

4.1.8  Agility 

Agility  is  one  of  the  most  important  characteristics  of  successful  Information  Age  organizations. 
Agility  is  the  ability  to  be  effective  in  changing,  nonlinear,  uncertain,  and  unpredictable 
environments.  Agile  organizations  are  the  result  of  an  organizational  structure,  command  and 
control  approach,  concepts  of  operation,  supporting  systems,  and  personnel  that  have  a 
synergistic  mix  of  the  right  characteristics.  The  term  agile  can  be  used  to  describe  each 
component  of  an  organization’s  mission  capability  packages,  and/or  an  organization  that  can 
instantiate  many  MCPs.  Since  agility  is  a  property  of  both  force  elements  and  C2  processes,  the 
lack  of  agility  in  one  or  more  of  these  components  will  affect  an  organization’s  overall  agility. 
Thus,  agile  C2  can  make  much  more  of  a  positive  difference  in  the  context  of  an  agile  force  than 
it  would  without  such  a  force.  However,  without  an  operational  concept  that  utilizes  agile  C2,  the 
agility  of  a  C2  system  (human  and  equipment)  will  have  only  modest  benefits.  Similarly,  an  agile 
force  that  does  not  have  an  agile  C2  system  and  operating  concepts,  cannot  perform  close  to  its 
capacity. 

Agile  forces,  MCPs,  C2  systems,  and  operating  concepts  make  sense  regardless  of  the  threat  or 
the  technology  environment.  However,  the  more  uncertain  and  dynamic  an  adversary  and/or  the 
environment  are,  the  more  valuable  agility  becomes.  Since  agility  is  a  property  that  is  manifested 
over  a  space  (a  range  of  values,  a  family  of  scenarios,  a  spectrum  of  missions)  and  time  rather 
than  being  associated  with  a  point  in  a  space  (e.g.  a  specific  circumstance,  a  particular  scenario,  a 
given  mission)  or  time,  agility  represents  capabilities  that  can  be  termed  scenario  independent. 
While  we  need  scenario  independence,  traditional  military  planning  is  threat-based  and  relies  on 
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a  few  likely  or  most  threatening  seenarios.  Threat-based  planning  arose  beeause  the  greatest 
threat  to  most  eountries  lay  in  one  or  more  hostile  neighbors  (for  example,  Iran  and  Iraq  in  the 
1980s).  The  key  to  designing  agile  C2  is  representing  the  diversity  of  threats  and  operating 
environments  in  a  way  that  samples  the  future  intelligently. 

Knowing  one’s  likely  adversaries  and  the  nature  of  their  forees,  military  establishments  eould 
study  their  likely  threats  and  design  speeifie  forees,  operational  eoneepts,  and  C2  systems  to 
eounteraet  them.  In  the  past,  arms  raees  have  provided  detailed  information  about  adversary 
eapabilities  and  intent,  leading  to  eounter  strategies  that  are  highly  speeialized  to  the  speeifie 
threat.  With  the  changing  strategic  environment,  however,  such  knowledge  of  adversarial 
capabilities  and  intentions  is  increasingly  difficult  to  obtain.  Rather,  U.S.  forces  must  prepare  for 
increased  uncertainty  and  ambiguity  in  the  future. 

Therefore,  agility  is  increasingly  becoming  recognized  as  the  most  critical  characteristic  of  a 
transformed  force,  with  net-centricity  being  understood  as  the  key  to  achieving  agility.  Military 
establishments  have  recognized  that  agility  considerations  must  permeate  a  mission  capability 
package,  not  just  be  considered  an  attribute  of  the  C2  system,  the  operational  concept,  or  the 
force.  This  implies  that  the  capability  to  be  agile  involves  having  not  only  the  right  materiel 
(sensors,  infostructure,  and  combat  systems)  but  also  the  right  doctrine,  organization,  personnel, 
training,  and  leadership.  Moreover,  it  implies  a  need  to  coevolve  these  MCP  elements  through 
experimentation  campaigns  that  assess  not  only  mission  effectiveness,  but  also  agility.  Indeed, 
coalition  partners  are  concerned  that  they  need  to  make  the  proper  near-term  investment 
decisions  in  order  to  keep  pace  with  U.S.  transformation.  The  potential  for  agility  is  greatly 
enhanced  by  the  shared  awareness  and  collaboration  in  Network  Centric  Operations.  In  essence, 
richer  information,  cognitive,  and  social  domains  enable  greater  agility. 

4.1.9  Agile  C2 

Agile  C2  only  makes  sense  in  the  context  of  agile  forces  and  operational  concepts.  Agile 
individuals  (commanders  for  example  may  differ  in  agility),  organizations,  C2  systems 
(personnel  plus  their  supporting  information  systems  and  decision  aids),  and  forces  have  a 
synergistic  combination  of  the  following  six  attributes,  the  key  dimensions  of  agility: 

1 .  Robustness:  Effectiveness  across  different  contexts  (the  ability  to  maintain  effectiveness 
across  a  range  of  tasks,  situations,  and  conditions). 

2.  Resilience:  Overcoming  losses,  damage,  setbacks  (the  ability  to  recover  from  or  adjust  to 
misfortune/damage,  and  the  ability  to  degrade  gracefully  under  attack  or  as  a  result  of 
partial  failure). 

3.  Responsiveness:  The  ability  to  react  to  a  change  in  the  environment  in  a  timely  manner. 

4.  Flexibility:  Multiple  ways  to  succeed  and  seamless  movement  between  them  (the  ability 
to  employ  multiple  ways  to  succeed  and  the  capacity  to  move  seamlessly  between  them). 

5.  Innovation:  The  ability  to  do  new  things  and  the  ability  to  do  old  things  in  new  ways. 

6.  Adaptation:  The  ability  to  change  work  processes  and  the  ability  to  change  the 
organization. 

While  these  attributes  of  agility  are  analytically  distinct  and  often  must  be  measured  in  different 
domains  and  contexts,  in  practice  they  are  often  interdependent.  Therefore,  when  one  of  these 


25 


attributes  is  lacking,  achieving  the  others  is  much  more  difficult.  However,  when  they  are  all 
present,  the  likelihood  of  success  (mission  accomplishment)  increases  greatly  (see  Figure  4-3 
below).  For  a  complete  discussion  of  the  attributes  of  Agility,  refer  to  Power  to  the  Edge}"^ 
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Figure  4-3.  The  Six  Aspects  of  Agility  in  the  Domains  of  Warfare 


4.2  Networking  and  Information 
4.2.1  Overview 

Networking  and  information  provide  the  foundation  for  Network  Centric  Operations.  The  tenets 
of  NCW,  as  reported  to  the  U.S.  Congress,'^  begin  with  the  statement:  “H  robustly  networked 
force  improves  information  sharing f  and  end  with:  “these  in  turn  dramatically  increase  mission 
effectiveness.'”  The  tenets  summarize  Network  Centric  Operations  as... improving  networking 
capabilities  in  order  to  increase  mission  effectiveness.  Thus,  networking  and  information  are 
central  to  Network  Centric  Operations.  Figure  4-4  below  highlights  Networking  and  Information 
in  the  NCO  metrics  framework. 


Alberts,  David  S.  &  Richard  E.  Hayes.  Power  to  the  edge:  command,  control  in  the  information  age.  Washington, 
DC:  CCRP  Publication  Series.  2003. 

Department  of  Defense.  Network  Centric  Warfare  Report  to  Congress.  July  2001. 
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Figure  4-4.  Networking  and  Information  in  the  NCO  Framework 


In  Power  to  the  Edge,  the  authors  state  that  as  bandwidth  beeomes  less  costly  and  more  widely 
available,  individuals  and  organizations  will  have  direct  and  simultaneous  access  to  information 
and  to  each  other  and  this  will  facilitate  their  ability  to  process  information  in  novel  ways.  The 
authors  describe  a  networked  collaborative  environment  as  one  that  “fully  enables  all  of  the 
attributes  of  reach,  richness,  and  quality  of  interactions,  allowing  the  utility  of  the  information 
exchange  to  be  significantly  increased,  helping  to  avoid  information  overload,  improve 
timeliness,  facilitate  collaboration,  and  create  the  conditions  for  self-synchronization.  These 
information-related  capabilities  are  all  enabled  by  the  post  and  smart  pull  approach  inherent  to  a 
robustly  networked  environment.”*’ 

Information  processing  occurs  in  a  rapidly  changing  environment  with  vast  amounts  of  data. 
The  ability  to  make  informed  decisions  in  such  an  environment  requires  that  the  available  data  be 
parsed  in  such  a  way  as  to  extract  only  the  specific  information  needed  in  real  time,  e.g.  smart 
pull.  The  intelligent  dissemination  of  data  in  a  rapid  manner  is  a  key  indicator  of  effective 
network  performance. 

Evaluating  network  performance  requires  an  understanding  of  what  network  effectiveness  means 
and  what  issues  must  be  considered.  Network  evaluation  requires  a  multiple-stakeholder 
perspective  and  occurs  at  three  broad  levels  of  analysis:  the  community,  the  network  itself,  and 
the  network’s  organizational  participant  levels.  While  overall  network  effectiveness  depends  on 
interactions  across  all  three  levels  of  analysis,  we  can  evaluate  each  level  separately.  Network 
effectiveness  at  the  community-level  is  judged  by  the  contribution  networks  make  to  the 


Power  To  The  Edge:  Command,  Control  In  The  Information  Age:  pg  xiv. 
Power  To  The  Edge:  Command,  Control  In  The  Information  Age:  pgs  81-82. 
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communities  they  serve.  A  network  must  satisfy  the  needs  and  expeetations  of  those  groups 

within  a  community  that  have  both  a  direct  and  indirect  interest  in  seeing  that  needs  are  met. 

Three  ways  of  evaluating  network-level  effectiveness  include:  the  ebb  and  flow  of  agencies  to 

and  from  the  network;  the  extent  to  which  services  that  are  aetually  needed  are  provided  by  the 

network;  and  the  strength  of  the  relationships  between  and  among  nodes.  Two  nodes  conneeted 

in  more  that  one  way  have  a  stronger  bond  than  two  nodes  connected  by  a  single  link.  Network 

effectiveness  at  the  organization/participant  level  is  based  on  four  criteria:  client  outcomes, 

1 8 

legitimacy,  resource  acquisition,  and  cost. 

For  example,  on  a  battlefield,  commanders  are  typieally  viewed  as  most  important,  and  satisfying 
their  needs  is  critical  for  success.  However,  commanders’  needs  are  likely  to  be  fragmented 
aeross  echelons,  resulting  in  multiple  constituent  groups  with  very  different  views  about  how 
sueeess  should  be  measured.  Division-level  commanders  may  have  one  view  of  what  the 
network  should  provide,  whereas  platoon-level  eommanders  with  different  requirements  may 
have  quite  a  different  view.  The  Conceptual  Framework  facilitates  measurement  of  network 
effectiveness  across  any  given  echelon,  functionality,  etc.  by  identifying  key  concepts,  attributes, 
and  metries. 

4.2.2  Networking 

In  information  technology,  a  network  is  a  series  of  points  or  nodes  interconnected  by 
communication  paths.  Networks  can  interconnect  with  other  networks  and  contain  sub¬ 
networks.  The  term  “networking”  ean  be  thought  of  as  an  intereonneetion  of  a  system  of 
eomputers,  eommunieations,  data  applieations,  seeurity,  people,  training,  and  other  support 
structures  that  provide  rapidly  and  intelligently,  local  and  global,  information  processing  and 
service  needs.  Rapid  and  intelligent  information  exchanges  are  timely  and  avoid  overload.  In  the 
NCO  metrics  framework.  Networking  eonsists  of  the  Network  (conneetivity  between  nodes)  and 
Net-Ready  Nodes  (foree  entities  capable  of  sharing  information  and  eollaborating  with  others). 
Hence,  networking  refers  to  the  extent  of  interconnection  among  force  entities.  The  Quality  of 
Networking  directly  depends  on  the  nodes  eomprising  the  network,  where  the  “nodes”  are  all  the 
foree  entities  eapable  of  sharing  information  and  eollaborating  with  others. 

Degree  of  networking  consists  of  three  attributes  that  describe  the  network.  The  first  attribute, 
reach,  addresses  the  degree  to  whieh  foree  entities  ean  interaet.  This  starts  with  physical 
connectivity,  and  extends  to  inelude  basie  interoperability  issues  that  enable  eontent  to  move 
among  entities.  Quality  of  service  focuses  on  the  extent  of  connectivity.  This  includes  the  type  of 
connection;  voice  or  text  data  to  full  video  conferencing  capabilities.  Network  Assurance 
addresses  the  eonfidenee  one  can  reasonably  have  that  force  entities  will  have  good  eonnectivity. 
This  ineludes  the  seeurity,  privaey,  and  integrity  of  the  network  and  its  contents. 

The  NCO  Conceptual  Framework  begins  by  defining  some  key  goals  for  the  network  and 
establishing  a  set  of  quantitative  metries  that  define  the  extent  to  whieh  the  goals  are  being 


**  Provan,  Keith  G.  &  H.  Brinton  Milward.  Do  Networks  Really  Work?  A  Framework  for  Evaluating  Public-Sector 
Organizational  Networks.  'Public  Administration  Review.  July/  August  2001.  Vol.  61.  No.  4:  pgs  414-423. 
http://searchnetworking.techtarget.com/sDefinition/0..sid7  gci212644.00.html 
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achieved.  The  metries  are  applied  to  the  network  to  determine  past  and  present  levels  of  goal 
attainment  and  incremental  changes  in  the  network.  Figure  4-5  below  illustrates  networking 
attributes  and  metrics. 


Degree  of  Networking 


Attribute 

Metrics 

Reach 

Percent  of  nodes  that  can  communicate  in  desired  access  modes, 
information  formats,  and  applications 

Quality  of  Service 

Vector  of  performance  metrics,  including  average  bandwidth  provided 
(available  and  bottleneck),  packet  delay,  delay  jitter,  and  data  loss 

Network 

Assurance 

Categorical  rating  from  “highly  secure”  to  “not  secure” 

(estimated  from  assessment  of  network’s  installed  security  software, 
hardware,  and  usage  policies) 

Figure  4-5.  Degree  of  Networking  Metrics 


The  Degree  of  Networking  measure  is  the  extent  to  which  force  entities  are  interconneeted,  or 
capable  of  being  interconnected.  Another  way  to  view  this  is  how  well  the  following  goals  are 
being  met: 

•  Make  information  available  on  a  network  that  people  depend  on  and  trust; 

•  Populate  the  network  with  new,  dynamic  sources  of  information  to  defeat  the  enemy;  and 

•  Deny  the  enemy  information  advantages  and  exploit  weakness  to  support  Network 
Centrie  Warfare  and  the  transformation  of  DoD  business  processes.^'’ 

4.2.3  Information 

Network  Centrie  Operations  postulates  that  effeetive  networking  leads  to  inereased  information 
sharing  and  ultimately  to  improved  force  effectiveness.  Networking  involves  much  more  than 
the  physical  communication  links  between  people  and  information  systems  that  they  use. 
Information  systems  in  NCO  must  produee  eoherent  information  that  ean  be  transformed  into 
awareness  and  then  understanding.  Because  information  exists  in  a  dynamic  environment, 
information  systems  must  have  the  ability  to  adjust  quickly  to  changing  requirements.  In  short, 
information  systems  must  produce  information  that  is  both  cohesive  and  flexible. 


http://www.defenselink.mil/nii/homepage.html 
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Information  is  stimulus  that  has  meaning  in  some  context  for  its  receiver.  When  information  is 
entered  into  and  stored  on  a  network  or  computer,  it  is  generally  referred  to  as  data.  After  some 
processing,  data  can  again  be  perceived  as  information.  The  metrics  framework  evaluates  three 
different  types  of  information;  organic,  individual,  and  shared  information. 

Joint  Publication  1-02  defines  “organic”  as  being  assigned  to  and  forming  an  essential  part  of  a 
military  organization.  Organic  information  is  information  that  is  “derived”  from  the  unit, 
community,  or  military  organization.  In  other  words,  organic  information  is  information  derived 
from  or  gathered  by  an  entity  that  is  not  shared  and  is  unavailable  to  the  network.  For  the  most 
part,  organic  information  remains  local  to  the  entity. 

The  attributes  for  organic  information  are  located  in  Figure  4-6.  They  include  Correctness, 
Consistency,  Currency,  and  Precision  and  have  a  long  history  of  applicability  in  the  evaluation  of 
command  center  performance  that  predates  the  origin  of  the  NCO  Conceptual  Framework.  The 
bottom  half  of  the  graphic  lists  specific  fitness-for-use  measures.  These  are  Completeness, 
Accuracy,  Relevance,  and  Timeliness.  These  two  sets  of  measures  are  directly  related.  For 
example,  correctness  of  a  perceived  enemy  unit’s  headquarters  location  can  be  measured  as  an 
error  in  meters  or  kilometers,  accompanied  by  some  clarification  concerning  the  specific  rules 
for  identifying  location.  Accuracy  is  also  measured  in  meters,  but  is  supplemented  by  some 
requirement  that  is  associated  with  the  sufficiency  for  use  of  the  information.  For  instance,  a  5km 
error  in  location  is  probably  sufficient  for  determining  general  direction  of  unit  movement  over 
time  in  support  of  operational  decisionmaking;  it  is  deficient  for  purposes  of  supporting  targeting 
activities. 


Quality  of  Organic  Information 


Attribute 

Definition 

Objective 

Measures 

Measures  quality  in  reference  to  criteria  that  are  independent  of  the 
situation 

Correctness 

Extent  to  which  information  is  consistent  with  ground  truth 

Consistency 

Extent  to  which  information  is  consistent  with  prior  information 

Currency 

Age  of  information 

Precision 

Level  of  measurement  detail  of  information  item 

Fitness  for  Use 
Measures 

Measures  quality  in  reference  to  criteria  that  are  determined  by  the 
situation 

Completeness 

Extent  to  which  information  relevant  to  ground  truth  is  collected 

Accuracy 

Appropriateness  of  precision  of  information  for  a  particular  use 

Relevance 

Proportion  of  information  collected  that  is  related  to  task  at  hand 

Timeliness 

Extent  to  which  currency  of  information  is  suitable  to  its  use 

Figure  4-6.  Quality  of  Organic  Information 


^’Department  of  Defense,  Dictionary  of  Military  and  Associated  Terms:  Joint  Publication  1-02,  12  April  2001:  pg 
391. 
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UAIW  speaks  of  networked  forees  in  whieh  entities  will  be  net-ready  to  eonneet,  with  the 
presumption  that  they  will  inereasingly  depend  upon  non-organic  information  for  their  preferred 
mode  of  operations.  Battlespace  entities  will  not  only  receive  information,  but  will  be  suppliers 

of  information  as  well.  Hierarchical  flows  of  information  will  be  streamlined,  and  peer-to-peer 

22 

flows  greatly  increased. 

Individual  Information  is  the  first  form  of  non-organic  information  that  entities  encounter. 
Individual  Information  refers  to  all  the  information  available  or  presented  to  an  entity.  Individual 
Information  provides  the  basis  for  awareness  and  understanding.  It  differs  from  Organic 
Information,  because  it  also  includes  information  that  has  been  distributed  over  a  network  and 
obtained  through  some  interaction.  The  attributes  for  Quality  of  Individual  Information  are  also 
present  in  Shared  Information  see  (Figure  4.8). 

Information  Shareability  refers  to  a  network’s  ability  to  accept,  index,  and  transmit  particular 
pieces  of  information,  including  data  elements,  data  files,  and  streams  of  information  quickly  and 
accurately.  Information  Shareability  is  only  concerned  with  whether  or  not  it  is  easy  to  make  data 
or  information  available  to  the  network,  and  whether  data  and  information  can  be  found  by  force 
entities.  It  only  considers  whether  or  not  what  is  submitted  to  the  network  is  indexed  correctly, 
stored  without  degradation,  transmitted  accurately  and  on  demand,  and  presented  to  the  receiver 
in  a  manner  equivalent  to  what  was  initially  submitted.  The  degree  of  Information  Shareability  is 
influenced  by  the  physical  properties  of  the  network,  including  the  transmission  speed,  accuracy, 
and  the  support  for  posting  and  retrieving  different  types  of  information.  Figure  4.7  lists  the 
attributes  of  Information  Shareability. 


Degree  of  Information  “Shareability” 


Attribute 

Definition 

Quantity  of  Posted 
Information 

Extent  to  which  collected  information  is  posted 

Quantity  of 
Retrievable 
Information 

Proportion  of  nodes  that  can  retrieve  various  sets  of  information. 
Determined  by  the  following: 

•Awareness  of  Information:  Degree  to  which  the  existence  of  the 
information  is  advertised  to  force  member 

•Access  to  Information:  Degree  to  which  access  to  information  is 
controlled 

•  Meta-data  of  Information:  Degree  to  which  information  has  labels 
describing  what  it  is  and  how  it  may  be  used  (facilitates  indexing  and 
searching) 

Ease  of  Use 

Degree  to  which  presentation  of  information  facilitates  desired  use 

Understanding  Information  Age  Warfare:  pg  295. 
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Figure  4-7.  Degree  of  Information  Shareahility 


Shared  Information  is  information  that  is  derived  from  the  network.  Note  that  the  attributes  for 
shared  information  are  similar  to  the  attributes  for  Individual  Information  with  one  exception;  the 
concept  of  extent.  This  attribute  measures  the  proportion  of  information  that  is  held  in  common 
across  force  entities.  Figure  4.8  below  lists  the  attributes  of  both  Individual  and  Shared 
Information.  The  Degree  of  Shared  Information  captures  both  the  quality  of  the  Shared 
Information,  and  the  extent  to  which  information  is  shared,  while  only  the  quality  of  the 
information  is  assessed  for  Individual  Information  and  Organic  Information. 


Degree  of  Shared  Information 


Attribute 

Definition 

Objective  Measures 

Measures  quality  in  reference  to  criteria  that  are  independent  of  the  situation 

Extent 

Proportion  of  information  in  common  across  force  entities,  within  and  across  communities 
of  interest  (Col) 

Proportion  of  force  entities  that  share  infonnation  item 

Correctness 

Extent  to  which  shared  information  is  consistent  with  ground  truth 

Consistency 

Extent  to  which  shared  information  is  consistent  within  and  across  Col 

Currency 

Age  of  shared  information 

Precision 

Level  of  measurement  detail  of  shared  information  item 

Quality 

Measures  quality  in  reference  to  criteria  that  are  determined  by  the  situation 

Completeness 

Extent  to  which  shared  infonnation  relevant  to  ground  truth  is  obtained 

Accuracy 

Appropriateness  of  precision  of  shared  infonnation  for  a  particular  use 

Relevance 

Proportion  of  shared  information  retrieved  that  is  related  to  task  at  hand 

Timeliness 

Extent  to  which  cunency  of  shared  information  is  suitable  to  its  use 

Figure  4-8.  Degree  of  Shared  Information 

Together,  Networking  and  Information  form  the  center  of  Network  Centric  Operations.  The 
extent  to  which  entities  are  networked  (quality  of  networking)  along  with  their  ability  to  rapidly 
and  intelligently  share  information  (quality  and  degree  of  information  sharing)  and  utilize  their 
organic  capabilities  (including  organic  information)  are  critical  in  determining  overall 
effectiveness.  Vice  Adm.  (ret.)  Arthur  K.  Cebrowski  asserted  that  a  "lessons  learned"  in  Iraq  is 
that  “good  sensors  networked  with  good  intelligence  and  disseminated  through  a  robust 

O’! 

networking  system  accelerate  combat  on  an  order  not  seen  before.” 


David  Hughes.  Networking,  Swarming  and  Warfighting.  Aviation  Week  &  Space  Technology,  September  29, 
2003:  pg  48.  http://www.oft.osd.mil/librarv/librarv  files/article  246  Aviation%20Weel.doe/ 
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4.3  Sensemaking:  Awareness,  Understanding,  and  Decisionmaking 


NCO  Conceptual  Framework 


Figure  4-8.  The  NCO  Conceptual  Framework 


As  depicted  within  the  NCO  Conceptual  Framework,  sensemaking  addresses  those  activities 
carried  out  at  both  the  individual  and  collaborative  level  to  (1)  “make  sense”  of  the  information 
available  in  the  context  of  experience/expertise,  (2)  draw  from  this  information  specific 
implications  regarding  potential  threats  and  opportunities  that  require  responsive  action,  and  (3) 
organize  these  inferences  into  actionable  knowledge  that  can  frame  key  decisions.  While  moving 
from  individual  sensemaking  to  shared  sensemaking  involves  interactions  among  different 
personnel  and  elements  of  the  C2  system,  this  aspect  of  the  process  is  addressed  separately 
within  the  NCO  CF  in  the  “Quality  of  Interactions”  top  level  concept  (see  section  4.4  below). 

Sensemaking  provides  the  link  between  the  information  domain  depicted  within  the  NCO 
framework  and  the  physical  domain  in  which  action  takes  place.  Sensemaking  is  largely  a 
cognitive  activity,  i.e.  it  takes  place  in  the  minds  of  individuals,  not  computers — that  is  strongly 
influenced  by  social  networks  and  social  interactions  at  the  collaborative  level.  This  is  why  we 
refer  to  sensemaking  as  a  “sociocognitive  activity.” 

Sensemaking  evolved  as  a  concept  from  the  earlier  OODA  loop  depicted  in  traditional  models  of 
C4ISR.  In  this  context,  sensemaking  most  accurately  corresponds  to  the  “orient”  step  in  the 
OODA  loop  model.  However,  as  thinking  has  progressed  regarding  the  OODA  loop  and  its 
application  to  Information  Age  warfare,  researchers  have  begun  to  demand  a  richer  set  of 
constructs  to  describe  the  process  of  turning  data  and  information  into  actionable  knowledge  and 
operational  decisions.  Thus,  in  the  book.  Understanding  Information  Age  Warfare,  the  “orient” 
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step  of  the  OODA  loop  model  was  expanded  to  refleet  a  hierarehieal  proeess  that  ineluded 
monitoring,  awareness,  understanding,  sensemaking,  eommand  intent,  battlespace  management, 
and  synehronization.  These  definitions  have  largely  remained  in  later  NCO  publications. 

Available  research  from  the  fields  of  naturalistic  decisionmaking,  management  science,  and 
complexity  theory  suggest  that  the  conceptualization  of  awareness,  understanding, 
decisionmaking,  and  so  forth,  should  be  merged  into  more  of  a  single  integrated  cognitive 
process  model.  As  our  conceptualization  of  these  processes  evolve  so  also  should  the 
corresponding  design  and  development  of  C4ISR  systems  supporting  this  cognitive  process.  The 
book.  Understanding  Information  Age  Warfare,  depicted  this  idea  in  a  series  of  charts  shown 
here  in  Figure  4-9.  The  traditional  model  of  C4ISR  shows  sensemaking  largely  separated  from 
monitoring  and  awareness  on  the  one  hand,  and  from  battlespace  management  on  the  other  hand. 
As  we  move  into  the  future,  these  processes  are  seen  to  merge  into  a  nearly  single  integrated 
process  that  extends  across  the  cognitive  domain. 


Figure  4-9.  Evolution  of  Process  Models 


The  concepts  of  awareness,  understanding,  and  decisions  are  briefly  defined  below.  Awareness 
is  a  process  state  existing  in  the  cognitive  domain.  That  is,  it  takes  place  in  the  minds  of  key 
leaders  and  their  supporting  battlestaffs,  not  in  computers.  Awareness  is  achieved  through  a 
complex  interaction  of  available  information,  e.g.  common  operational  picture,  with  prior 
knowledge  and  beliefs  representing  the  experience  and  expertise  of  the  battlestaff.  Awareness 
relates  to  the  operational  situation  as  it  currently  is  or  was  in  the  past.  By  contrast, 
understanding  is  defined  as  the  process  state  of  drawing  inferences  about  possible  consequences 
of  the  operational  situation.  It  is  based  on  the  ability  of  the  battlestaff — acting  individually  and 
collaboratively — ^to  predict  possible  future  patterns  of  the  battlespace.  That  is,  whereas 
awareness  deals  with  the  battlespace  as  it  was  or  is,  understanding  deals  with  the  battlespace  as  it 
is  becoming.  Interpreting  these  patterns — spatially,  functionally,  temporally — in  the  context  of 
the  goals/objectives,  constraints,  and  planned  courses  of  action  envisioned  for  the  operation,  the 
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battlestaff  begins  to  identify  potential  threats  and  opportunities  that  demand  a  responsive  ehange 
or  deeision  from  the  eommand  authorities. 

Decisions  involve  volition  and  ehoiees  about  what  is  to  be  done  to  ensure  that  the  mission  is  on  a 
eourse  that  will  aehieve  the  intended  goals/objeetives.  But  notiee  deeisions  only  take  plaee 
within  the  “deeision  spaee”  defined  by  the  battlestaff  s  understanding.  Henee,  in  the 
sensemaking  model,  understanding  is  seen  to  be  the  key  step  in  the  proeess,  whereas  deeisions 
are  seen  as  simply  the  eulminating  ratilieation  of  the  mental  aetivities  that  produeed 
understanding.  However,  deeisions  involve  something  beyond  merely  understanding:  they  refleet 
eommitment  of  the  military  organization  to  aetion. 

While  good  deeisions  are  generally  based  upon  good  understanding,  this  is  not  always  the  ease. 
In  faet,  some  deeisions  are  foreed  or  initiated  by  events  within  the  battlespaee — with 
understanding  being  developed  in  a  post  hoe  manner.  Likewise,  awareness  ean  be  influeneed  by 
understanding — i.e.  attention  to  partieular  details  of  the  battlespaee  might  be  triggered  by 
adopting  a  speeifie  set  of  hypotheses  as  to  what  an  adversary  might  be  attempting.  The  point  here 
is  that  the  proeesses  of  awareness,  understanding,  and  deeisions  do  not  always  flow  in  one 
direetion,  but,  rather,  ean  unfold  in  a  very  interaetive,  emergent  manner.  Nevertheless,  however, 
these  proeesses  represent  different  aspeets  of  sensemaking  that  ean  be  isolated,  measured,  and 
assessed. 

The  importanee  of  developing  a  eorreet  understanding  of  a  situation  is  supported  by  empirieal 
researeh  evidenee  that  eorrelates  quality  of  understanding  with  the  quality  of  subsequent 
deeisions  that  emerge  from  the  C2  proeess.  This  evidenee  euts  aeross  the  taetieal  and 
operational  levels  of  eommand.  While  it  is  reeognized  that  understanding  refleets  a  great  deal  of 
subjeetivity — i.e.  it  is  based  on  interpretations  and  hypotheses  developed  from  prior 
experienee — it  is  possible  to  define  “eorreetness”  or  “quality”  in  terms  of  how  experieneed 
subjeet  matter  experts  might  size  up  a  given  situation.  Thus,  in  terms  of  measurement  and 
assessment,  quality  of  understanding  ean  be  ealibrated  in  terms  of  senior  offioers  who  advise  or 
eritique  a  partieular  operation  or  exereise.  Inereasingly,  “ground  truth”  can  be  obtained  via 
“instrumented  reality”  exercises  and  simulations,  making  assessment  of  correctness  more 
reliable.^^ 

Sensemaking  is  both  an  individual  process  and  a  collective  process.  At  the  individual  level, 
sensemaking  involves  the  mental  relating  of  situation  understanding  with  action.  That  is,  it 
involves  building  a  mental  model  that  hypothesizes  how  a  situation  might  evolve  over  time,  what 
threats  and  opportunities  for  action  are  likely  to  emerge  from  this  evolution,  what  potential 
actions  can  be  taken  in  response,  what  are  the  projected  outcomes  of  those  responses,  and  what 


Enhancements  to  Army  Command  and  Control  Evaluation  System  (ACCES)  Draft  Technical  Report,  October 
1990  finds  that  Quality  of  Situational  Awareness  is  the  best  predictor  of  Command  and  Control  Effectiveness  (C2), 
Force  Effectiveness,  and  Military  Plan  Quality.  Also  during  the  1990s,  Air-to-Air  Mission:  Offensive  and  Defensive 
Counter  using  Voice  +  Data  Link  Improved  Situational  Awareness  which  led  to  a  150%  improvement  in  operations 
(see  Understanding  Information  Age  Warfare:  pg  252). 

For  example,  the  Future  Combat  System  C2  Prototype  Experimentation  conducted  by  the  Army’s  CECOM  and 
DARPA  collects  extensive  data  on  ground  trath,  making  it  relatively  simple  to  compare  the  commander’s  articulated 
understanding  of  the  battle  space  with  ground  truth. 
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cultural  values  drive  the  ehoice  of  future  action.  This  part  of  the  process  involves  three 
interrelated  activities:  (1)  generating  alternative  response  actions  to  control  the  situation;  (2) 
identifying  the  objectives,  constraints,  and  factors  that  influence  the  feasibility  and  desirability  of 
each  alternative,  and  (3)  conducting  an  assessment  of  these  choiees.  As  noted  earlier,  these  three 
processes  are  often  integrated  into  a  single  mental  activity  and  can  be  either  (1)  the  subject  of  a 
very  formal  staff  process  or  (2)  as  simple  as  one  officer  examining  a  situation  and  making  up  his 
mind. 

At  the  collective  level,  sensemaking  is  represented  as  a  collaborative  process  involving  different 
functional  perspectives  and  possible  stakeholder  interests.  Military  operations  involve  the 
coordination  of  many  different  functional  elements — eaeh  of  whieh  will  be  “seeing”  specific 
emerging  threats  and  opportunities  from  their  own  perspective.  Different  stakeholder  interests 
might  also  exist  across  organizational  boundaries  within  a  coalition.  In  order  to  achieve 
operational  synehronization,  these  perspectives  must  be  melded  into  a  common  problem 
framework  where  the  different  aspects  of  the  operation  are  adjudicated  and  integrated  into  a 
single  vision.  Thus,  shared  sensemaking  becomes  a  crueial  part  of  the  C4ISR  proeess. 

The  fact  that  sensemaking  occurs  at  a  collective  level  implies  the  need  to  add  a  fourth  domain  to 
the  NCO  framework:  the  social  domain  where  different  force  entities  interact.  Adding  the  social 
domain  to  the  NCO  framework  brings  with  it  a  new  set  of  factors  and  issues.  Whether  or  not 
these  different  entities  can  effectively  exehange  information,  reconcile  perspectives,  and  achieve 
a  eommon  understanding  and  vision  of  the  operation  depends  not  only  on  the  technical  means  of 
communicating  and  collaborating,  but  also  on  various  social  factors  that  govern  this  collective 
process.  Shared  sensemaking  involves  human  interaction  and  collaboration  across  different 
communities  of  expertise  and  cultural  boundaries.  Commonality  of  training,  doetrine,  and 
procedures  affect  the  proeess  of  shared  sensemaking,  as  do  the  factors  of  interpersonal 
familiarity  and  trust  and  overall  organizational  leadership. 

We  now  move  into  a  discussion  of  the  aetual  NCO  framework  section  on  sensemaking,  as 
depicted  in  the  next  set  of  slides  that  summarize  the  measures  and  metries  developed  to  date. 
Figure  4-10  defines  the  various  attributes  of  individual  awareness.  As  seen  here,  these  attributes 
include  both  objective  measures  (measures  of  quality  that  are  independent  of  the  operational 
situation)  and  fitness  of  use  measures  (measures  that  are  contextually  defined  in  terms  of  the 
demands  of  a  speeific  situation). 

Highlighted  are  two  examples  that  illustrate  the  general  approach  taken  for  observing  and 
assessing  attributes  of  individual  awareness.  At  the  top  of  the  table,  we  see  that  awareness  is 
objectively  assessed  in  terms  of  correctness,  consistency,  currency,  and  precision.  “Currency,” 
for  example,  reflects  the  time  lag  of  awareness  of  an  individual.  This  type  of  measure  is 
considered  objective  in  the  sense  that  it  is  independent  of  a  particular  decision  event  or  emergent 
threat/opportunity.  Figure  4-11  represents  how  it  might  be  calculated. 
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Attribute 

Metrics 

Objective  Measures 

Measures  quality  in  reference  to  criteria  that  are  independent  of  the  situation 

Correctness 

Correspondence  with  gronnd  trnthcorrelation  coefficient  (0=  no  convergence,  l=full 
convergence  between  individuals  awareness  and  gronnd  truth) 

Consistency 

Degree  of ‘deviation’  from  awareness  gained  from  previous  time  period 

Currency 

Time  lag  of  awareness 

Precision 

Level  of  granularity  ofawareness 

Fitness  for  Use 

Measnres 

Measures  quality  in  reference  to  criteria  that  are  determined  by  the  situation 

Compieteness 

Percentage  of  ground  truth  picture  included  in  awareness 

Accuracy 

Degree  to  which  precision  matches  what  is  needed  (0=no  match,  10=high  degree  of  matching 
between  precision  level  needed  and  available) 

Relevance 

Proportion  of  awareness  that  is  related  to  task  at  hand 

Timeliness 

Degree  to  which  currency  matches  what  is  needed  (0=no  match,  10=high  degree  of  matching 
between  currency  level  needed  and  available) 

Uncertainty 

Confidence  level  (0%  =uncertain,  100%=  certain)  or  confidence  interval  (95%,  90%,  etc.)  of 
awareness 

Figure  4-10.  Individual  Awareness:  Attributes  and  Metrics 
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Figure  4-1 1.  Currency  of  Awareness 


By  contrast,  moving  to  the  bottom  of  the  table,  we  see  that  “timeliness”  is  a  fitness-for-use 
measure.  That  is,  “timeliness”  refleets  the  degree  to  which  the  currency  of  the  information 
eomprising  awareness  suitably  supports  the  use  of  this  awareness  for  building  understanding  and 
making  deeisions.  In  other  words,  “timeliness”  expresses  the  degree  to  whieh  the  eurreney  of 
awareness  provides  an  adequate  window  of  decisionmaking  opportunity  for  the  battlestaff. 

With  metries,  the  goal  is  to  define  each  attribute  in  terms  that  support  quantitative  measurement 
and  eomparison  between  different  eases.  For  example,  it  might  be  possible  in  some  situations  to 
measure  the  degree  of  statistieal  eorrelation  of  awareness  with  ground  truth  that  exists  aeross  a 
relevant  set  of  objects  or  features  within  the  battlespaee. 
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As  a  fitness-for-use  measure,  timeliness  will  involve  the  judgment  of  subject  matter  experts  to 
assess  the  degree  to  which  the  timeliness  of  individual  awareness  meets  the  requirements  of  a 
specific  operational  situation.  Such  judgments  can  be  quantitatively  captured  using  a  metric 
approach  based  on  Likert  scales. 

By  comparison,  when  we  move  to  an  assessment  of  shared  awareness,  our  focus  is  now  placed 
on  a  comparison  of  perceptions  across  different  individuals — or,  in  some  cases,  between 
different  functional  staff  elements  or  between  different  organizations.  Thus,  to  the  set  of 
objective  measures  we  add  the  measure  of  “extent.”  Figure  4-12  provides  a  list  of  all  the 
attributes  and  their  associated  metrics  for  shared  awareness. 


Attribute 

Metrics 

Objective  Measures 

Measures  quality  in  reference  to  criteria  that  are  independent  of  the  situation 

Extent 

Percentage  of  awareness  elements  (relevant  objects  and  events)  held  in  common  within  and 
across  each  Community  of  Interest 

Correctness 

Correspondence  with  ground  truth-correlation  coefficient  (0=  no  convergence,  l=full 
convergence  between  individual’s  awareness  and  ground  truth) 

Consistency 

Degree  of  ‘deviation’  from  awareness  gained  from  previous  time  period 

Currency 

Time  lag  of  awareness 

Precision 

Level  of  granularity  of  awareness 

II  Hill  i\  i  ' 

Fitness  for  Use 

Measures 

Measures  quality  in  reference  to  criteria  that  are  determined  by  the  situation 

Completeness 

Percentage  of  ground  truth  picture  included  in  awareness 

Accuracy 

Degree  to  which  precision  matches  what  is  needed  (0=no  match,  10=high  degree  of  matching 
between  precision  level  needed  and  available) 

Relevance 

Proportion  of  awareness  that  is  related  to  task  at  hand 

Timeliness 

Degree  to  which  currency  matches  what  is  needed  (0=no  match,  10=high  degree  of  matching 
between  currency  level  needed  and  available) 

Uncertainty 

Confidence  level  (0%  =uncertain,  100%=  certain)  or  confidence  interval  (95%,  90%,  etc.)  of 
awareness 

Figure  4-12.  Shared  Awareness :  Attributes  and  Metrics 

Here,  extent  is  a  measure  used  to  reflect  the  degree  to  which  different  force  elements  develop 
and  maintain  a  common  awareness  within  and  across  specific  communities  of  interest  (sets  of 
battlestaff  elements  and  force  entities  involved  in  synchronizing  specific  aspects  of  the 
operation).  As  seen  here,  “extent”  can  be  measured  in  two  ways:  (1)  comparing  the  commonality 
of  awareness  across  force  entities  and  (2)  assessing  the  proportion  of  awareness  held  in  common 
across  force  entities. 

Again,  moving  to  the  level  of  quantifiable  metrics,  we  see  that  extent  of  shared  awareness  might 
be  measured  utilizing  a  measure  such  as  Cronbach’s  alpha.  Figure  4-13  presents  an  illustrative 
example  of  how  this  could  be  calculated. 


38 


List  of  Key 
Battlespace 
Objects  & 
Activities 

Ni 

N2 

C2  Node 

N3  N4  .. 

N, 

Oi 

1 

1 

1 

0 

1 

O2 

1 

0 

0 

0 

1 

O3 

0 

1 

1 

1 

1 

Oj 

0 

1 

1 

1 

1 

El 

1 

0 

1 

0 

0 

E2 

1 

1 

1 

0 

1 

E3 

1 

1 

0 

0 

0 

E. 

1 

0 

0 

0 

1 

Cronbach’s  Alpha 


a  =i 


1  +(i-1)  ■  r 


where?  is  the  average  intemode 
correlation  among  key  objects 
and  activities 


Figure  4-13.  Extent  of  Shared  Awareness 


For  assessing  individual  understanding,  the  NCO  framework  employs  a  set  of  measures  and 
metrics  that  are  consistent  with  those  used  for  assessing  awareness  (see  Figure  4-14  below). 
However,  given  the  increased  complexity  of  understanding  and  its  greater  dependence  upon 
personal  experience  and  expertise,  these  measures  and  metrics  will  be  interpreted  differently 
from  those  used  for  assessing  awareness. 


Attribute 

Definition 

Objective  Measures 

Measures  quality  in  reference  to  criteria  that  are  independent  of  the  situation 

Correctness 

Extent  to  which  understanding  is  consistent  with  gronnd  trnth 

Consistency 

Extent  to  which  understanding  is  internally  consistent  with  prior  understanding 

Currency 

Time  lag  of  nnderstanding 

Precision 

Level  of  grannlarity  of  understanding 

Fitness  for  Use  Measures 

Measures  quality  in  reference  to  criteria  that  are  determined  by  the  situation 

Completeness 

Extent  to  which  relevant  nnderstanding  is  obtained 

Accuracy 

Appropriateness  of  precision  of  nnderstandingfor  a  particular  use 

Relevance 

Proportion  ofunderstanding  obtained  by  force  member  that  is  related  to  task  at  hand 

Timeliness 

Extent  to  which  currency  of  understandingis  snitable  to  its  use 

Uncertainty 

Subjective  assessment  of  confidence  in  nnderstanding 

Figure  4-14.  Individual  Understanding:  Attributes  and  Metrics 


Using  again  the  example  of  “currency,”  Figure  4-15  below  illustrates  how  it  might  be  calculated 
for  individual  understandings. 
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Figure  4-15.  Currency  of  Understanding 

A  similar  pattern  of  attribute  measures  is  shown  here  for  shared  understanding.  As  with 
awareness,  a  measure  defined  as  “extent”  of  shared  understanding  is  added  to  the  list  of  measures 
to  reflect  the  degree  to  which  a  common  understanding  is  held  within  and  across  specific 
Communities  of  Interest.  See  Figure  4-16  for  a  complete  list  of  all  attributes  and  metrics  for 
shared  understanding. 


Attribute 

Definition 

Objective  Measures 

Measures  quality  in  reference  to  criteria  that  are  independent  o  f  the  situation 

Extent 

Proportion  of  understanding  in  common  across  force  entities,  within  and  across  Communities 
of  Interest  (COI) 

Proportion  of  force  entities  that  share  a  given  nnderstanding 

Correctness 

Extent  to  which  shared  nnderstanding  is  consistent  with  ground  truth 

Consistency 

Extent  to  which  shared  nnderstanding  is  consistent  within  and  across  COI 

Currency 

Time  lag  of  shared  understanding 

Precision 

Level  of  grannlarity  of  shared  understanding 

Quality 

Measures  quality  in  reference  to  criteria  that  are  determined  by  the  situation 

Completeness 

Extent  to  which  relevant  shared  nnderstanding  is  obtained 

Accuracy 

Appropriateness  of  precision  of  shared  nnderstanding  for  a  particular  use 

Relevance 

Proportion  of  shared  understanding  that  is  related  to  task  at  hand 

Timeliness 

Extent  to  which  cnrrency  of  shared  understanding  is  suitable  to  its  use 

Uncertainty 

Subjective  assessment  of  confidence  in  shared  nnderstanding 

Figure  4-16.  Shared  Decisions:  Attributes  and  Metrics 


Finally,  we  come  to  the  decision  component  of  sensemaking.  Figure  4-17  lists  all  attributes  and 
metrics  for  individual  decisionmaking. 
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Attribute 

Metrics 

Objective  Measures 

Measures  quality  in  reference  to  criteria  that  are  independent  of  the  sitnation 

Consistency 

Extent  to  which  decisions  are  internally  consistent  with  prior  understanding  and  decisions 

Currency 

Time  lag  of  decisions 

Precision 

Level  of  granularity  of  decisions 

Fitness  for  Use  Measures 

Measures  quality  in  reference  to  criteria  that  are  determined  by  the  situation 

Appropriateness 

Extent  to  which  decisions  are  consistent  with  existing  nnderstanding,  command  intent  and 
values 

Completeness 

Extent  to  which  relevant  decisions  encompass  the  necessary: 

•Depth:  range  of  actions  and  contingencies  included 
•Breadth:  range  of  force  elements  included 
•Time:  range  of  time  horizons  included 

Accuracy 

Appropriateness  of  precision  of  decisions  for  a  particular  use 

Relevance 

Proportion  of  decisions  that  are  significant  to  task  at  hand 

Timeliness 

Extent  to  which  currency  of  decision  making  is  suitable  to  its  use 

Uncertainty 

Subjective  assessment  of  confidence  in  decisions 

Risk  Propensity 

Extent  of  risk  aversion 

Mode  of  Decision  Making 

Type  of  decision  making  process  utifized  (naturalistic,  dominated,  min-max,  expected 
utility) 

Figure  4-17.  Individual  Understanding:  Attributes 


Two  examples  illustrate  the  nature  of  assessment  at  this  level.  In  the  first  example,  we  see  that 
deeision  time  can  be  measured  both  objectively  and  in  terms  of  fitness-for-use.  Objectively,  we 
can  measure  the  time  lag  of  decisions — say,  from  when  an  adversary  initiates  a  specific 
operation  to  when  the  C2  system  formulates  a  specific  response  decision.  Figure  4-18  represents 
the  currency  of  decisionmaking. 


Time  Time 

of  Event 

Event  Posted 


T 


event 


post 


Time 

Time 

Time 

Event 

Information 

Time 

Event 

Response 

Received 

Event 

Significance 

Declared 

at 

Noticed 

Projected 

by 

Command  Ctr 

by  Staff 

by  Staff 

Commander 

T 


receive 


T 


awareness 


T 


understanding 


T 


decision 


Time  lag  of  decision 

Figure  4-18.  Currency  of  Decisionmaking 
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The  second  example  highlights  the  measure  of  “appropriateness.”  Here,  “appropriateness” 
reflects  the  degree  to  which  individual  decisions  are  consistent  with  (1)  the  existing 
understanding  of  the  situation,  (2)  command  intent,  and  (3)  the  values  of  the  military 
organization.  As  with  other  measures  of  this  degree  of  complexity  and  situational  dependence, 
assessment  of  this  measure  would  be  based  on  senior  subject  matter  expertise. 

Following  the  same  pattern  of  moving  from  individual  awareness  and  understanding  to  shared 
awareness  and  understanding,  we  see  that  collaborative  decisions  involve  the  additional  measure 
of  “extent,”  defined  here  as  the  proportion  of  force  entities  effectively  involved  in  reaching  a 
collaborative  decision  (see  Figure  4-19). 

In  addition,  the  definitions  of  other  measures  are  expanded  to  reflect  the  shared  nature  of  the 
process.  Thus,  for  example,  appropriateness  of  collaborative  decisions  is  measured  with  respect 
to  the  degree  it  reflects  shared  understanding,  command  intent,  and  shared  team  or  organizational 
values. 

Finally,  it  is  noted  that  the  degree  of  synchronization  of  decisions  across  force  elements  is  not 
addressed  in  this  portion  of  the  NCO  framework.  Synchronization,  an  important  element  of  NCO, 
is  addressed  next  as  its  own  separate  area  of  measurement  and  assessment. 


Attribute 

Definition 

Objective  Measures 

Measures  quality  in  reference  to  criteria  that  are  independent  of  the  situation 

Extent 

Proportion  of  force  entities  that  reach  a  collaborative  decision 

Consistency 

Extent  to  which  decisions  are  in  agreement  across  force  entities,  within  and  across  COI 

Currency 

Time  lag  of  decisions 

Precision 

Level  of  granularity  of  decisions 

Fitness  for  Use  Measures 

Measures  quality  in  reference  to  criteria  that  are  determined  by  the  situation 

Appropriateness 

Extent  to  which  decisions  are  consistent  with  existing  shared  understanding,  command  intent  and 
shared  team  values 

Completeness 

Extent  to  which  relevant  decisions  encompass  the  necessary: 

•Depth:  range  of  actions  and  contingencies  included 
•Breadth:  range  of  force  elements  included 
•Time:  range  of  time  horizons  included 

Accuracy 

Appropriateness  of  precision  of  decisions  for  a  particular  use 

Relevance 

Proportion  decisions  that  are  important  to  the  accomplishment  of  the  task  at  hand 

Timeliness 

Extent  to  which  currency  of  decision  making  is  suitable  to  its  use 

Uncertainty 

Inter-subjective  assessment  of  confidence  in  decisions 

Risk  Propensity 

Extent  of  risk  aversion 

Mode  of  Decision  Making 

Type  of  collaborative  decision  making  structure  utilized  (authoritative  decision  making,  consensus 
building,  majority  rule,  etc.) 

Figure  4-19.  Collaborative  Decisions:  Attributes  and  Metrics 
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4.4  Quality  of  Interactions 


NCO  Conceptual  Framework 


Figure  4-20.  The  NCO  Conceptual  Framework 


4.4.1  Role  of  “Quality  of  Interactions”  in  the  Conceptual  Framework 

Interactions  involve  force  entities  actively  sharing  information,  and  developing  awareness, 
understanding  and/or  making  decisions  (developing  plans)  in  a  collaborative  fashion  while 
working  together  toward  a  common  purpose.  The  “Quality  of  Interactions”  top-level  concept 
bridges  the  gap  between  individual  and  shared  information  and  sensemaking  and  cuts  across  the 
Information,  Cognitive  and  Social  Domains  in  the  Conceptual  Framework.  The  concept  “Quality 
of  Interactions”  also  includes  “control  variables”  that  impact  performance  and  effectiveness. 
Importantly,  it  provides  a  means  to  measure  the  extent  to  which  network-centric  processes  are 
being  implemented.  As  Mission  Capability  Packages  co-evolve,  we  expect  that  the 
characteristics  and  behaviors  of  individuals  and  organizations  will  reflect  these  changes.  The 
attributes  of  Quality  of  Interaction  are  indicators  of  the  key  process  elements  that  are  likely  to 
evolve  as  we  move  toward  a  network-centric  force. 

Depending  on  the  context,  the  quality  of  interactions  can  be  considered  an  independent  variable 
or  a  dependent  variable.  As  an  independent  (or  explanatory)  variable,  it  can  impact  or  influence 
the  values  of  Individual  and/or  Shared  Sensemaking  directly  and  the  degree  of 
decision/actions/entities  synchronized  and  the  degree  of  effectiveness  indirectly.  For  instance,  an 
experimental  design  may  be  established  to  explore  how  different  modes  of  interaction  (reflecting 
different  MCPs)  impact  shared  awareness  and  understanding.  Holding  constant  all  the  other 
variables  in  the  NCO  Value  Chain  that  precede  Quality  of  Interactions  in  the  Conceptual 
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Framework  (like  degree  of  networking),  one  could  determine  how  different  modes  or  types  of 
collaboration  impact  decisionmaking. 

As  a  dependent  variable,  researchers  would  primarily  be  interested  in  how  net-centric 
technologies  and  practices  impact  the  ways  in  which  people  interact  and  collaborate.  Although 
mission  effectiveness  is  always  of  interest  and  concern,  it  may  be  practical  to  limit  an  experiment 
and/or  analysis  to  a  subset  of  the  NCO  CF.  Factors  that  impact  the  quality  of  interactions  are 
information  shareability,  individual  and  shared  information  and  sensemaking.  The  quality  of 
networking  has  an  indirect  impact  on  quality  of  interactions.  If  we  want  to  isolate  the  impact  of 
NCO  Value  Chain  Concepts,  it  is  important  to  include  control  variables  that  capture  exogenous 
characteristics  of  the  interactions. 

In  many  cases,  the  quality  of  interactions  can  be  considered  an  intervening  variable,  that  is,  it  is 
an  intermediate  factor  that  mediates  the  impact  of  net-centric  technologies  and  practices  on 
effectiveness.  In  such  cases,  if  we  wish  to  isolate  the  impact  of  NCO  variables  on  high-level 
outcomes  such  as  effectiveness,  it  is  essential  to  include  the  control  factors  mentioned  previously. 

4. 4. 2  Models  of  Interaction 

The  model  of  interactions  presented  in  the  Conceptual  Framework  represents  efforts  to  integrate 
existing  knowledge  on  teamwork,  collaboration,  and  interaction  in  military  and  non-military 
contexts.  It  draws  on  and  extends  existing  theories  and  research  on  team  effectiveness, 
collaboration,  and  complex  multiteam  interactions.  Three  models  of  interactions  are  particularly 
relevant  to  the  model  developed  here.  The  Team  Effectiveness  Model  developed  by  Ed  Salas,  of 
the  University  of  Central  Elorida,  et  ah;  the  Collaboration  Effectiveness  Model,  developed  by 
David  Noble  of  EBR,  which  focuses  on  the  factors  that  are  necessary  for  successful 
collaboration;  and  the  Multi-Team  Effectiveness  Model  by  John  Mathieu  of  University  of 
Connecticut,  which  provides  insights  into  interactions  that  cut  across  individual  organizations, 
echelons,  services,  function,  etc.  These  interactions  are  best  thought  of  as  episodes  of  multiteam 
collaboration.  This  approach  emphasizes  the  interdependencies  of  interactions  over  time  and 
space.  We  discuss  each  of  these  models  below. 

The  model  of  Team  Effectiveness  represented  in  Eigure  4-21  was  originally  developed  by  Salas 
et  al  and  has  been  widely  adopted  and  adapted  in  the  field  of  team  effectiveness  research.  This 
model  assumes  that  team  effectiveness  is  a  function  of  inputs  (such  as  individual,  team  and  task 
characteristics),  and  processes,  such  as  communication  and  coordination.  Outcomes  are 
measured  in  terms  of  products  of  the  collaborative  process  (meeting  objectives/mission  success), 
and  are  typically  measured  by  the  quantity  and  quality  of  the  products  and  the  efficiency  and 
effectiveness  of  the  process  involved  in  producing  those  products. 
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Figure  4-21.  Models  of  Interaction  (1) 


David  Noble,  of  EBR,  has  developed  a  Collaboration  Effectiveness  Model  that  focuses  in  detail 
on  the  specific  knowledge  processes  and  behaviors  that  facilitate  successful  collaboration.  As 
indicated  in  Eigure  4-22  below,  this  model  explicitly  ties  collaboration  to  individual  and  shared 
sensemaking.  Dr.  Noble  distinguishes  among  activities  aimed  at  setting  up  and  adjusting  the 
team,  group  problem  solving  activities,  and  activities  focused  on  synchronization  and  actions. 
There  are  multiple  feedback  loops  between  interactions  and  sensemaking,  illustrating  the 
complex  dependencies  among  these  activities. 

Collaboration  Effectiveness  Model  (Noble,  2003) 
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Figure  4-22.  Models  of  Interaction  (2) 


Recent  work  by  John  Mathieu  et  al  provides  a  useful  extension  of  the  Input-Process-Output 
model  of  team  effectiveness  that  illustrates  key  insights  relevant  to  Information  Age  interactions. 
The  authors  argue  that  most  activities  of  interest  involve  multiple  “teams”  that  cut  across 
echelons,  organizations,  countries,  etc.,  and  time  and  space.  They  extend  the  IPO  model  to 
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include  multiple  tasks,  eompeting  goal  hierarehies  and  eomplex  dependeneies  across  Inputs, 
Processes,  and  Outcomes.  The  distinguishing  characteristic  of  Multi-Teams  is  that  they  typically 
are  from  different  organizations,  echelons,  etc.,  and  they  have  different  proximate  goals, 
identities,  and  praetices,  but  they  share  a  common  distal  goal  (command  intent).  They  also 
exhibit  input,  proeess,  and  outeome  interdependeneies.  Figure  4-23  illustrates  a  stylized  Seenario 
in  which  multiple  teams  across  services  and  coalition  partners  operate  to  achieve  tasks  that  have 
complex  dependencies  across  time  and  space. 


Multi-Team  Effectiveness  Model  (Mathieu,  Marks,  zaecaro,  2000) 


4. 4. 3  NCO  Conceptual  Framework  Model  of  Interactions 

Building  on  these  three  models  and  other  theory  and  research,  the  NCO  CF  model  of  interactions 
identifies  the  key  inputs,  processes  and  outcomes  of  interaetions.  At  the  outcome  level,  we 
diverge  from  the  models  that  measure  effects  in  terms  of  the  produets  of  interaetion  and  instead 
focus  on  the  quality  of  interactions  themselves,  and  not  the  results.  We  do  this  because  the 
accomplishment  of  tasks/missions  is  measured  elsewhere  in  the  conceptual  framework.  Instead, 
we  focus  on  the  depth,  breadth,  intensity  and  agility  of  interaetions.  We  also  diverge  from  the 
Team  Effeetiveness  Model  by  using  the  term  “organization”  rather  than  “team.”  Building  on  the 
multiteam  work,  we  assume  that  activities  of  interest  can  range  from  traditional  teams  to  systems 
of  multiteams,  to  full  fledged  organizations.  We  designate  all  of  these  interactions  as 
“organizations.” 

Individual  and  Organizational  characteristics  are  highlighted  as  important  faetors  that  impaet  the 
quality  of  interactions  and  are  equivalent  to  the  INPUTS  in  the  previous  models.  Organizational 
and  Individual  Behaviors  also  impact  the  quality  of  interactions  and  are  equivalent  to  the 
PROCESSES  of  previous  models.  In  the  eontext  of  the  NCO  CE,  sometimes  these  inputs  and 
proeesses  ean  be  manipulated,  i.e.  aspects  of  MCPs  that  can  be  systematieally  varied;  and 
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sometimes  they  ean  be  considered  to  be  “control  variables”  in  that  they  are  outside  of  the  theory 
of  NCO  but  nonetheless  can  be  important  explanatory  factors  of  mission  effectiveness. 

Figure  4-24  is  a  representation  of  the  model  of  interactions  developed  in  the  Conceptual 
Framework 


The  focus  of  interaction:  share  information,  develop  and  share  awareness, 
develop  and  share  understandings,  make  decisions 
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Figure  4-24.  Quality  of  Interactions:  Attributes  and  Exogenous  Variables 


.  There  are  four  “top-level”  attributes  of  interactions: 

■  Depth  -  Measures  that  describe  the  nature  of  the  substance  of  interactions 

■  Breadth  -  Measures  that  describe  the  force  entities  that  interact 

■  Intensity  -  Measures  that  describe  the  pace  and  completeness  of  interactions 

■  Agility  -  Measures  that  that  describe  the  Robustness,  Resilience,  Flexibility, 
Responsiveness,  Innovativeness,  and  Adaptability  of  interactions 


Each  of  the  four  primary  attributes  of  Quality  of  Interactions  is  composed  of  sub-attributes. 
Figure  4-25  presents  each  of  the  four  top  level  attributes  and  all  of  the  associated  sub-attributes. 
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Attribute 

Definition 

Depth 

Measures  that  describe  the  nature  of  the  substance  of  interactions 

Quantity 

The  quantity  of  information,  awareness,  understandings,  and/or  decisions  that  are  the 
focus  of  interactions 

Quality 

The  quality  of  information,  awareness,  understandings,  and/or  decisions  that  are  the 
focus  of  the  interactions 

Breadth 

Measures  that  describe  the  force  entities  that  interact 

Reach 

The  number  of  members  that  participate  in  the  interactions 

Selectivity 

The  ability  to  reach  a  selected  subet 

Intensity 

Measures  that  describe  the  pace  and  completeness  of  interactions 

Continuity 

The  persistence  of  the  exchange  among  members  (continuous  to  episodic) 

Synchronicity 

Type  of  interaction:  synchronous  or  asynchronous  in  time  and  space 

Mode 

Degree  to  which  all  senses  are  involved  (ranges  from  face  to  face  with  data  +  voice  to 
voice  or  data  onlv) 

Latency 

The  time  lag  of  interactions 

Asiiilz _ 

Robustness,  Resilience,  Flexibilitv,  Responsiveness,  Iimovativeness,  and  Adaptability 

Figure  4-25.  Quality  of  Interactions :  Top  Level  Attributes 


We  will  focus  on  the  quantity  of  interactions  to  illustrate  why  this  is  a  useful  measure  of  overall 
quality  of  interactions.  If  we  consider  the  Air-to-Air  Case  Study,  it  was  hypothesized  that  one 
explanation  for  the  dramatic  increase  in  effectiveness  was  that  pilots  with  Link  16  plus  Voice 
communications  used  their  time  differently  than  pilots  with  Voice  Only.  Figure  4-26  illustrates 
this.  It  illustrates  how  the  quantity  of  interactions  and  the  nature  of  those  interactions  may  be 
impacted  by  network  centric  technologies  and  practices. 

Hypotheses: 

I.  Information  sharing  via  Voice  +  Link  1 6  leads  to  less  time  necessary  to  gather  critical 
information,  which  results  in  more  time  available  for  pilots  and  crew  to  interact  sharing 
awareness  and  understandings 

II.  Information  sharing  via  Voice  +  Link  1 6  leads  to  less  time  necessary  for  wingman  to 
gather  and  monitor  critical  information,  which  results  in  more  time  available  for 
wingman  to  interact  sharing  awareness  and  understandings 


Bll  (Flight  lead) 


B12  (Wingman) 


- ► 

Time 

Figure  4-26.  Quality  of  Interactions 
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Hypotheses  I  states  that  with  Link  16  plus  Voiee  communieations,  pilots  are  spending  mueh  less 
time  interaeting  with  the  erew  to  gather  information.  Instead,  the  foeus  of  interaetions  shifts 
away  from  exehanging  information  to  building  and  sharing  awareness  and  understandings. 
Hypothesis  11  extends  this  reasoning  to  the  wingman.  Instead  of  foeusing  on  information 
gathering,  monitoring  and  reporting,  the  wingman  ean  spend  mueh  more  time  sharing  awareness 
and  understandings. 

Although  this  is  hypothetieal,  it  is  quantifiable  with  the  NCO  CF  attributes  and  metries.  In  order 
to  gather  data  neeessary  to  test  this  hypothesis,  we  could  use  cockpit  recordings  and  with 
qualified  coders  determine  the  nature  of  the  interactions  among  and  between  crew  members. 

As  stated  above,  a  basic  assumption  is  that  characteristics  and  behaviors  of  individuals  and 
organizations  have  an  impact  on  the  likelihood  of  successful  interactions  (that  is,  they  impact  the 
quality  of  interactions  obtainable).  These  factors  are  discussed  next. 

4.4.4  Individual  Characteristics 

The  individual  characteristics  that  we  focus  on  are  risk  propensity,  competency,  trust, 
organizational  identification,  and  confidence  (see  Figure  4-27  below).  We  will  focus  on  one  of 
these  attributes,  the  Organizational  Identity  of  the  individual,  to  illustrate  the  approach.  Research 
on  identification  indicates  that  interactions  in  which  the  individuals  involved  identify  with  the 
group  (or  collective),  that  is  they  share  important  goals  and  values,  are  more  likely  to  be  of  a 
higher  quality  both  in  terms  of  the  subjective  assessments  of  the  participants  and  in  terms  of  the 
outcomes  (or  products)  of  the  interactions .  Within  a  military  context,  this  relates  to  loyalty  and 
commitment  to  command  intent.  In  the  Air-to-Air  example,  because  these  were  training  missions 
we  can  assume  that  organizational  identification  did  not  vary  between  the  voice  only  crews  and 
the  Link  16  crews.  However,  if  our  case  study  involves  multiple  teams  across  services,  agencies, 
coalitions,  etc.,  we  would  expect  that  people’s  identification  with  the  goals  and  norms  of  the 
organization  would  vary  dramatically.  In  fact,  in  these  cases,  we  would  expect  competing  goals 
and  values.  The  Conceptual  Framework  provides  the  means  to  measure  the  extent  to  which  this 
occurs  and  provides  a  mechanism  to  determine  the  impact  this  has  on  effectiveness. 


Jehn,  K.,  Chadwick,  C.,  and  Thatcher,  S.M.B.  (1997).  "To  Agree  or  Not  to  Agree:  The  Effects  of  Value 
Congruence,  Individual  Demographic  Dissimilarity,  and  Conflict  on  Group  Outcomes."  International  Journal  of 
Conflict  Management,  8,  287-305. 
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Attributes 

Definition 

Risk  Propensity 

Extent  of  risk  aversion 

Competence 

Level  of  knowledge,  skills,  abilities,  and  attitudes  (KSAAs) 

Trnst 

Extent  to  which  individual  is  willing  to  rely  on  other  members 

Organizational 

Identiilcation 

Extent  to  which  individual’s  identities  align  with  organizational  identities 

Confidence 

Degree  of  individual’s  expectation  that  other  members  are  reliable 

Figure  4-27.  Exogenous  Variables:  Individual  Characteristics 

4.4.5  Team/Organization  Characteristics 

In  addition  to  the  characteristics  of  the  individual,  the  characteristics  of  the  organization  (or  team 
or  collection  of  teams)  matters  as  well.  We  have  identified  eleven  attributes  that  influence 
mission/objective  success:  risk  propensity,  competence,  trust,  confidence,  size,  hardness, 
diversity,  permanence,  autonomy,  structure  and  interdependence  (see  Figure  4-28  below). 


Attributes 

Definitions 

Risk  Propensity 

Extent  of  risk  aversion 

Competence 

Distribution  of  members  knowledge,  skills,  abilities  and  attitudes  (KSAAs) 

Trust 

Extent  to  which  members  are  willing  to  rely  on  one  another 

Confidence 

Extent  to  which  members  have  expectations  of  the  reliability  of  the  organization 

Size 

Number  of  team  members  involved 

Hardness 

Degree  to  which  team  members  have  interacted  in  the  past  on  the  same  task 

Diversity 

Degree  to  which  team  members  are  heterogeneous  or  homogeneous  across  exogenous 
variables:  experience,  age,  gender,  etc. 

Permanence 

Expected  duration  of  organization 

Autonomy 

Extent  to  which  organization  is  externally  or  self  directed 

Stmcture 

Distribution  of  peer  and  authority  relationships 

•  Layers  of  authority 

•  Functional  Differentiation 

•  Connectedness  within  and  across  layers 

•  Directness  of  connections 

Interdependence 

Extent  to  which  members  depend  on  one  another  for  resources  (materials,  KSAAs,  etc. 

Figure  4-28.  Exogenous  Variables:  Organizational  Characteristics 


Interdependence  is  an  illustrative  attribute.  It  is  defined  to  be  the  extent  to  which  members 
depend  on  one  another  for  resources.  In  the  Air-to-Air  example,  crew  members  could  meet  their 
resource  needs  independently.  They  relied  on  one  another  and  the  AWACs  for  information  but 
their  actions  did  not  require  coordination  of  inputs  and  processes.  If  we  take  the  stylized  SEAD 
scenario  presented  earlier,  however,  we  can  see  that  in  some  cases  there  is  a  high  degree  of 
interdependence  among  members.  This  is  illustrated  in  Figure  4-29  below.  This  illustrates  the 
complex  interdependences  that  can  exist  for  a  given  mission.  Recall  that  these  elements  are 
typically  from  different  services  and  coalition  partners.  In  this  case,  the  demands  on  the  members 
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are  quite  high  in  terms  of  eoordination  of  actions.  We  would  expect  different  interaction  patterns 
to  emerge  with  different  levels  of  interdependence.  We  believe  it  plausible  that  this  could  impact 
the  effectiveness  of  net  centric  technologies  and  practices.  Therefore,  gathering  information  on 
the  nature  of  the  dependencies  is  important  if  we  are  to  evaluate  the  impact  of  net-centric 
technologies  and  practices  on  outcomes. 


Figure  4-29.  Organizational  Characteristics:  Interdependence 


4. 4. 6  Organizational  and  Individual  Behaviors 

Organizational  and  Individual  Behaviors  are  processes  that  impact  the  quality  of  interactions. 
These  include  the  level  of  cooperation,  the  efficiency  of  the  interactions,  how  synchronized  are 
the  interactions,  how  engaged  are  the  participants  and  the  extent  to  which  members  spend  their 
time  and  efforts  on  performing  tasks  versus  maintaining  the  team  (see  Figure  4-30  below). 


Attributes 

Definitions 

Cooperation 

Extent  to  which  member(s)  are  willing  and  able  to  work  together 

Efficiency 

Extent  to  which  members  utilize  one  another’ s  resources  so  as  to  minimize  costs  and 
maximize  benefits 

Synchronization 

Extent  to  which  organization  is  conflicted,  deconflicted,  or  synergistic 

Engagement 

Extent  to  which  all  members  actively  and  continuously  participate 

Team  vs.  Task  Balance 

Extent  to  which  efforts  are  directed  to  organizational  issues  vs.  relating  to  the  objective 

Figure  4-30.  Exogenous  Variables:  Organizational  and  Individual  Behaviors 

The  attribute  Engagement  is  useful  to  illustrate  how  it  is  an  appropriate  measure  of  quality  of 
interactions.  In  the  Air-to-Air  case  study,  it  was  hypothesized  that  one  explanation  for  the 
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increased  kill  ratio  was  the  fact  that  the  wingman  could  actively  engage  in  attacks,  rather  than 
passively  gathering,  monitoring  and  reporting  information. 

Interaetions  involve  foree  entities  actively  sharing  information,  and  developing  awareness, 
understanding  and/or  making  deeisions  (developing  plans)  in  a  eollaborative  fashion  while 
working  together  toward  a  eommon  purpose.  In  the  Conceptual  Framework,  Quality  of 
Interactions  bridges  the  gap  between  individual  and  shared  information  and  sensemaking  and 
euts  across  the  Information,  Cognitive  and  Social  Domains.  It  includes  important  “eontrol 
variables”  that  impact  performance  and  effectiveness,  such  as  individual  and  organizational 
characteristics  and  behaviors.  It  is  a  crucial  intermediate  step  between  network  centric 
technologies  and  outeomes.  It  is  essential  to  have  a  way  to  measure  the  extent  to  whieh  network 
centric  processes  are  being  implemented.  The  Quality  of  Interaetions  attributes  and  metrics 
provides  this. 


52 


5.0  Summary 

The  NCO  Conceptual  Framework  represents  the  latest  innovations  in  the  theory  of  Network 
Centric  Operations.  It  builds  on  the  original  tenets  of  NCW  and  extends  them  in  significant 
ways:  Agility  is  introduced  as  a  key  concept  of  Network  Centric  Operations;  the  social  domain  is 
explicitly  introduced;  and  the  distinction  between  individual  and  shared  sensemaking  is 
highlighted.  The  NCO  CF  “top-level”  extends  and  elaborates  the  key  concepts  of  the  tenets  and 
develops  those  concepts  that  mediate  between  changes  in  network  centric  technologies  and 
practices  and  mission  effectiveness.  It  provides  a  means  to  capture  evidence  on  the  co-evolving 
elements  of  MCPs  that  result  from  changes  in  network  centricity.  The  NCO  CF  presents  a  rich 
set  of  attributes  and  metrics  for  each  top-level  concept,  providing  the  means  for  researchers  and 
analysts  to  gather  evidence  on  NCO  related  activities.  This  facilitates  evaluation  of  progress 
toward  NCO  and,  importantly,  facilitates  answering  the  key  question  “what  is  the  impact  of  NCO 
on  mission  outcomes?”  The  NCO  CF  helps  us  to  answer  the  “why”  question,  that  is,  it  provides 
the  means  to  explain  the  dramatic  increases  in  effectiveness  that  are  being  reported  when 
network  centric  technologies  and  practices  are  adopted.  The  Air-to-Air  case  study  provides  an 
important  initial  validation  of  the  NCO  CF.  As  the  NCO  CF  is  applied  to  more  case  studies  and 
utilized  in  experiments  and  other  applications,  it  is  anticipated  that  it  will  continue  to  mature  and 
develop;  contributing  to  a  growing  knowledge  base  on  NCO  related  data  and  evidence  that  can 
be  used  in  the  effort  to  transform  the  DoD. 

The  transformation  of  the  DoD  from  an  industrial  to  an  information  age  organization  is 
underway.  The  Office  of  Force  Transformation  is  leading  this  effort,  in  part  by  conducting  and 
sponsoring  cutting  edge  research  on  transformation  related  technologies  and  practices.  Network 
Centric  Operations  is  a  crucial  element  of  transformation  and  is  the  focus  of  this  document.  The 
OFT’s  support  of  the  Network  Centric  Operations  Conceptual  Framework  Program  is  intended  to 
mature  the  theory  of  network  centric  operations,  refine  and  further  develop  the  Conceptual 
Framework  by  applying  it  enterprise  wide  (experimentation,  active  engagements,  case  studies, 
and  short  courses).  This  document,  the  Draft  Network  Centric  Operations  Conceptual 
Framework  Version  1.0,  is  intended  to  summarize  the  current  version  of  the  NCO  Conceptual 
Framework.  It  is  the  draft  version  of  the  first  of  two  formal  versions  of  the  Conceptual 
Framework  expected.  As  the  Conceptual  Framework  evolves  over  the  course  of  the  NCO  CF 
Program,  future  versions  of  this  document  will  capture  those  changes. 
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